Aug 04 16:01:37.352502 ip-10-0-132-42 systemd[1]: Starting Kubernetes Kubelet... Aug 04 16:01:37.687822 ip-10-0-132-42 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 16:01:37.687822 ip-10-0-132-42 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 16:01:37.687822 ip-10-0-132-42 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 16:01:37.687822 ip-10-0-132-42 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 16:01:37.687822 ip-10-0-132-42 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 16:01:37.689465 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.689393 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 16:01:37.696177 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696158 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 16:01:37.696177 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696173 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 16:01:37.696177 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696178 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 16:01:37.696177 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696181 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 16:01:37.696177 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696184 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696187 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696191 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696194 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696196 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696200 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696203 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696206 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696208 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696211 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696214 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696230 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696234 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696238 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696241 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696243 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696246 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696249 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696251 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696254 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 16:01:37.696382 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696256 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696259 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696261 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696264 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696268 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696272 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696275 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696284 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696286 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696289 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696291 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696294 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696297 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696299 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696302 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696305 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696308 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696317 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696320 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 16:01:37.696939 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696323 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696326 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696328 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696330 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696333 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696335 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696338 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696341 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696343 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696346 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696348 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696351 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696353 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696356 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696358 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696361 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696365 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696368 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696371 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 16:01:37.697419 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696374 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696376 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696380 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696408 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696412 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696415 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696418 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696421 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696425 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696427 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696430 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696433 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696435 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696438 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696441 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696444 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696447 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696449 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696451 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696454 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 16:01:37.697863 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696456 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 16:01:37.698359 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696460 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 16:01:37.698359 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696463 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 16:01:37.698359 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696465 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 16:01:37.698359 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696468 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 16:01:37.698359 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696470 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 16:01:37.698359 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696473 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 16:01:37.698359 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696475 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 16:01:37.698359 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696478 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 16:01:37.698359 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696480 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 16:01:37.698359 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696483 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 16:01:37.698359 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696485 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 16:01:37.698359 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696487 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 16:01:37.698359 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696490 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 16:01:37.698359 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696492 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 16:01:37.698359 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696495 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 16:01:37.698359 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.696498 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698629 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698637 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698641 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698644 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698647 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698649 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698652 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698654 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698657 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698659 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698662 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698664 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698667 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698669 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698672 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698674 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698677 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698680 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698682 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 16:01:37.698766 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698685 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698687 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698689 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698692 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698694 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698697 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698699 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698701 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698705 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698708 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698712 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698714 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698717 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698720 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698724 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698727 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698730 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698734 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698736 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 16:01:37.699259 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698739 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698741 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698744 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698747 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698749 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698752 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698754 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698756 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698759 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698761 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698764 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698767 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698770 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698773 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698776 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698779 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698781 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698784 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698786 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 16:01:37.699767 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698789 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698791 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698794 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698797 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698800 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698802 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698806 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698809 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698813 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698816 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698819 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698821 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698824 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698826 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698829 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698832 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698834 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698837 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698839 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 16:01:37.700207 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698841 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698844 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698846 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698848 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698851 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698853 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698855 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698858 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698861 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698863 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698866 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698868 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698871 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698873 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698876 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698878 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698881 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698884 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698886 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698889 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 16:01:37.700662 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698891 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698895 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.698898 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699244 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699250 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699253 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699255 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699258 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699260 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699263 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699266 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699268 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699271 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699274 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699276 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699279 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699282 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699284 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699287 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699291 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 16:01:37.701132 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699295 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699298 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699301 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699303 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699306 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699309 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699311 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699313 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699316 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699319 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699321 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699324 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699326 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699329 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699332 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699334 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699337 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699339 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699341 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699344 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 16:01:37.701606 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699346 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699348 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699351 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699354 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699356 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699359 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699361 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699363 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699366 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699368 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699371 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699373 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699377 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699380 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699383 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699386 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699389 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699392 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699395 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 16:01:37.702072 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699398 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699400 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699403 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699405 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699408 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699411 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699413 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699416 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699426 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699430 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699433 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699436 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699438 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699441 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699443 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699446 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699448 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699451 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699454 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699456 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 16:01:37.702530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699459 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699461 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699464 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699466 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699469 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699471 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699474 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699477 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699479 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699482 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699487 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699489 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699492 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699494 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699497 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699500 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699502 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699504 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699507 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699510 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 16:01:37.702998 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699512 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699514 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.699517 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699583 2580 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699590 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699595 2580 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699599 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699603 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699606 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699610 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699614 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699617 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699620 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699623 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699626 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699629 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699632 2580 flags.go:64] FLAG: --cgroup-root="" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699635 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699637 2580 flags.go:64] FLAG: --client-ca-file="" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699644 2580 flags.go:64] FLAG: --cloud-provider="external" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699647 2580 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699650 2580 flags.go:64] FLAG: --cluster-domain="" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699653 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699656 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 16:01:37.703486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699659 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699662 2580 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699666 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699669 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699672 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699674 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699677 2580 flags.go:64] FLAG: --contention-profiling="false" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699680 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699683 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699686 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699688 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699694 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699698 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699700 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699703 2580 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699708 2580 flags.go:64] FLAG: --enable-server="true" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699710 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699715 2580 flags.go:64] FLAG: --event-burst="100" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699718 2580 flags.go:64] FLAG: --event-qps="50" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699721 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699723 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699726 2580 flags.go:64] FLAG: --eviction-hard="" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699730 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699732 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699735 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 16:01:37.704075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699738 2580 flags.go:64] FLAG: --eviction-soft="" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699741 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699743 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699747 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699750 2580 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699753 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699757 2580 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699760 2580 flags.go:64] FLAG: --feature-gates="" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699763 2580 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699767 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699770 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699773 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699776 2580 flags.go:64] FLAG: --healthz-port="10248" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699779 2580 flags.go:64] FLAG: --help="false" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699782 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-132-42.ec2.internal" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699785 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699788 2580 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699791 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699794 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699797 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699800 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699803 2580 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699807 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 16:01:37.704693 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699810 2580 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699813 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699816 2580 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699819 2580 flags.go:64] FLAG: --kube-reserved="" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699821 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699824 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699827 2580 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699830 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699832 2580 flags.go:64] FLAG: --lock-file="" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699835 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699838 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699841 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699846 2580 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699849 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699852 2580 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699856 2580 flags.go:64] FLAG: --logging-format="text" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699859 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699862 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699865 2580 flags.go:64] FLAG: --manifest-url="" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699867 2580 flags.go:64] FLAG: --manifest-url-header="" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699871 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699874 2580 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699878 2580 flags.go:64] FLAG: --max-pods="110" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699881 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699884 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 16:01:37.705303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699887 2580 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699890 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699893 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699895 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699898 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699905 2580 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699909 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699911 2580 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699914 2580 flags.go:64] FLAG: --pod-cidr="" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699917 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699922 2580 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699925 2580 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699928 2580 flags.go:64] FLAG: --pods-per-core="0" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699930 2580 flags.go:64] FLAG: --port="10250" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699933 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699936 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0768d1e3e483ba050" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699939 2580 flags.go:64] FLAG: --qos-reserved="" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699942 2580 flags.go:64] FLAG: --read-only-port="10255" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699945 2580 flags.go:64] FLAG: --register-node="true" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699947 2580 flags.go:64] FLAG: --register-with-taints="" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699951 2580 flags.go:64] FLAG: --registry-burst="10" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699954 2580 flags.go:64] FLAG: --registry-qps="5" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699957 2580 flags.go:64] FLAG: --reserved-cpus="" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699962 2580 flags.go:64] FLAG: --reserved-memory="" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699966 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 16:01:37.705913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699969 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699972 2580 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699975 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699978 2580 flags.go:64] FLAG: --runonce="false" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699980 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699983 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699986 2580 flags.go:64] FLAG: --seccomp-default="false" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699989 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699992 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699995 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.699998 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700001 2580 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700004 2580 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700008 2580 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700011 2580 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700013 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700016 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700019 2580 flags.go:64] FLAG: --system-cgroups="" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700022 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700027 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700029 2580 flags.go:64] FLAG: --tls-cert-file="" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700032 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700036 2580 flags.go:64] FLAG: --tls-min-version="" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700038 2580 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700041 2580 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 16:01:37.706523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700044 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700046 2580 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700049 2580 flags.go:64] FLAG: --v="2" Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700053 2580 flags.go:64] FLAG: --version="false" Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700057 2580 flags.go:64] FLAG: --vmodule="" Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700061 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.700064 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700164 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700168 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700171 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700174 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700177 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700180 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700183 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700185 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700188 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700190 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700193 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700196 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700198 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700202 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 16:01:37.707100 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700204 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700207 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700210 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700213 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700228 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700230 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700233 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700236 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700238 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700241 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700244 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700246 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700249 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700252 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700254 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700257 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700262 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700264 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700267 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700269 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 16:01:37.707609 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700272 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700275 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700277 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700280 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700283 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700285 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700288 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700290 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700293 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700297 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700300 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700305 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700307 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700310 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700313 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700316 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700320 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700324 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 16:01:37.708078 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700328 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700331 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700334 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700337 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700340 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700342 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700345 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700347 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700350 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700352 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700356 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700359 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700361 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700364 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700366 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700369 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700371 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700374 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700376 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700379 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 16:01:37.708584 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700381 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700384 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700386 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700389 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700391 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700395 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700398 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700400 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700403 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700405 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700408 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700410 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700413 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700415 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700418 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700420 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700423 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700425 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700428 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700430 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 16:01:37.709046 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700432 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 16:01:37.709530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700435 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 16:01:37.709530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700439 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 16:01:37.709530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700441 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 16:01:37.709530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700444 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 16:01:37.709530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700447 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 16:01:37.709530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.700449 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 16:01:37.709530 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.701058 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 16:01:37.709530 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.707101 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 16:01:37.709530 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.707115 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 16:01:37.709530 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.707842 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 16:01:37.709530 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.707850 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 16:01:37.709530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707904 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 16:01:37.709530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707908 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 16:01:37.709530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707911 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 16:01:37.709530 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707915 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707917 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707921 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707924 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707926 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707929 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707931 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707934 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707937 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707939 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707942 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707946 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707949 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707953 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707955 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707958 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707960 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707962 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707965 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707968 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 16:01:37.709889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707971 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707973 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707976 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707979 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707981 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707984 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707987 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707989 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707992 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707994 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707997 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.707999 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708001 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708004 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708006 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708009 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708011 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708013 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708016 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 16:01:37.710374 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708018 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708021 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708023 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708026 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708028 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708031 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708033 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708036 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708038 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708041 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708043 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708045 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708049 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708053 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708055 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708058 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708061 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708063 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708065 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 16:01:37.710803 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708069 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708073 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708076 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708079 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708081 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708084 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708087 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708089 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708092 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708094 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708097 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708099 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708102 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708104 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708107 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708109 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708112 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708114 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708117 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708119 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 16:01:37.711251 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708121 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708124 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708126 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708128 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708131 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708133 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708136 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708138 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708141 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708144 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708146 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708148 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708151 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708153 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708160 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708162 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708165 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 16:01:37.711752 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708167 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 16:01:37.712144 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.708172 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 16:01:37.712144 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708286 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 16:01:37.712144 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708291 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 16:01:37.712144 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708294 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 16:01:37.712144 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708296 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 16:01:37.712144 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708300 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 16:01:37.712144 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708302 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 16:01:37.712144 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708305 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 16:01:37.712144 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708308 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 16:01:37.712144 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708311 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 16:01:37.712144 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708313 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 16:01:37.712144 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708316 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 16:01:37.712144 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708318 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 16:01:37.712144 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708321 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 16:01:37.712144 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708323 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708326 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708328 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708331 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708333 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708337 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708341 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708344 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708347 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708350 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708353 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708355 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708359 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708362 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708365 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708369 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708371 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708374 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 16:01:37.712520 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708377 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708379 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708382 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708385 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708387 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708390 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708392 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708395 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708397 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708400 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708402 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708405 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708407 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708409 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708412 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708414 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708417 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708419 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708422 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 16:01:37.712940 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708424 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708427 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708429 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708431 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708434 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708436 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708439 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708441 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708444 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708446 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708448 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708451 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708454 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708456 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708458 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708461 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708463 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708466 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708468 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708470 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 16:01:37.713461 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708473 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708475 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708478 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708480 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708485 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708487 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708489 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708493 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708496 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708498 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708501 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708503 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708506 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708508 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708511 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708513 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708516 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708518 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708521 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708523 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 16:01:37.713919 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708526 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 16:01:37.714386 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708528 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 16:01:37.714386 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708531 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 16:01:37.714386 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708533 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 16:01:37.714386 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708536 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 16:01:37.714386 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708539 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 16:01:37.714386 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708541 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 16:01:37.714386 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708544 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 16:01:37.714386 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:37.708546 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 16:01:37.714386 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.708551 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 16:01:37.714386 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.708643 2580 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 16:01:37.714386 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.710711 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 16:01:37.714386 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.711879 2580 server.go:1019] "Starting client certificate rotation" Aug 04 16:01:37.714386 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.711977 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 16:01:37.714386 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.712017 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 16:01:37.714727 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.714341 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 16:01:37.720168 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.720150 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 16:01:37.727778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.727763 2580 log.go:25] "Validated CRI v1 runtime API" Aug 04 16:01:37.731364 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.731346 2580 log.go:25] "Validated CRI v1 image API" Aug 04 16:01:37.732137 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.732122 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 16:01:37.734329 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.734307 2580 fs.go:135] Filesystem UUIDs: map[177f9861-c6b3-4ad6-8fbf-aeaa7ec26ac8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ddd572a8-b9b7-4f5d-b03b-8406a8158db2:/dev/nvme0n1p3] Aug 04 16:01:37.734390 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.734329 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 16:01:37.739208 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.739111 2580 manager.go:216] Machine: {Timestamp:2026-08-04 16:01:37.737956636 +0000 UTC m=+0.285352087 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200378 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24635cefe58a1d59ae33370b66ed44 SystemUUID:ec24635c-efe5-8a1d-59ae-33370b66ed44 BootID:74304957-69b7-4ae2-96e0-1a9393fa076c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cf:dc:63:19:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cf:dc:63:19:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:7a:b3:15:c5:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 16:01:37.739208 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.739203 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 16:01:37.739317 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.739304 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 16:01:37.742192 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.742163 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 16:01:37.742321 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.742192 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-42.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 16:01:37.742842 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.742833 2580 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 16:01:37.742870 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.742845 2580 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 16:01:37.742870 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.742852 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 16:01:37.743526 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.743515 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 16:01:37.744093 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.744076 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 16:01:37.744594 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.744583 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 16:01:37.748414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.748401 2580 state_mem.go:36] "Initialized new in-memory state store" Aug 04 16:01:37.748523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.748513 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 16:01:37.748554 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.748541 2580 kubelet.go:486] "Attempting to sync node with API server" Aug 04 16:01:37.748554 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.748549 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 16:01:37.748617 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.748559 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 16:01:37.748617 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.748569 2580 kubelet.go:398] "Adding apiserver pod source" Aug 04 16:01:37.748617 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.748578 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 16:01:37.748854 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.748839 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 16:01:37.748934 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.748863 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 16:01:37.751167 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.751152 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 16:01:37.752386 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.752370 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 16:01:37.752477 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.752408 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 16:01:37.753350 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.753338 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 16:01:37.753416 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.753357 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 16:01:37.753416 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.753366 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 16:01:37.753416 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.753375 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 16:01:37.753416 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.753388 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 16:01:37.753416 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.753396 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 16:01:37.753416 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.753407 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 16:01:37.753416 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.753415 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 16:01:37.753636 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.753425 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 16:01:37.753636 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.753433 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 16:01:37.753636 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.753457 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 16:01:37.753636 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.753466 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 16:01:37.754086 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.754076 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 16:01:37.754136 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.754088 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 16:01:37.757274 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.757259 2580 server.go:1268] "Started kubelet" Aug 04 16:01:37.757357 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.757336 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 16:01:37.757468 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.757427 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 16:01:37.757548 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.757481 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 16:01:37.757959 ip-10-0-132-42 systemd[1]: Started Kubernetes Kubelet. Aug 04 16:01:37.762379 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.761103 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 16:01:37.762379 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:37.762064 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-42.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 16:01:37.762379 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:37.762161 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 16:01:37.762379 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.762271 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-42.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 16:01:37.764079 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.764051 2580 server.go:310] "Adding debug handlers to kubelet server" Aug 04 16:01:37.764343 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:37.764323 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 16:01:37.766441 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:37.765565 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-42.ec2.internal.18c8a591c5a5fce6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-42.ec2.internal,UID:ip-10-0-132-42.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-42.ec2.internal,},FirstTimestamp:2026-08-04 16:01:37.757232358 +0000 UTC m=+0.304627781,LastTimestamp:2026-08-04 16:01:37.757232358 +0000 UTC m=+0.304627781,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-42.ec2.internal,}" Aug 04 16:01:37.769110 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.769084 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 16:01:37.769110 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.769102 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 16:01:37.769259 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:37.769167 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-42.ec2.internal\" not found" Aug 04 16:01:37.769610 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.769593 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 16:01:37.769610 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.769596 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 16:01:37.769742 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.769621 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 16:01:37.769791 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.769750 2580 reconstruct.go:97] "Volume reconstruction finished" Aug 04 16:01:37.769791 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.769761 2580 reconciler.go:29] "Reconciler: start to sync state" Aug 04 16:01:37.769938 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.769923 2580 factory.go:55] Registering systemd factory Aug 04 16:01:37.769986 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.769940 2580 factory.go:223] Registration of the systemd container factory successfully Aug 04 16:01:37.770162 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.770147 2580 factory.go:153] Registering CRI-O factory Aug 04 16:01:37.770253 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.770165 2580 factory.go:223] Registration of the crio container factory successfully Aug 04 16:01:37.770253 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.770208 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 16:01:37.770253 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.770244 2580 factory.go:103] Registering Raw factory Aug 04 16:01:37.770408 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.770272 2580 manager.go:1196] Started watching for new ooms in manager Aug 04 16:01:37.770646 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:37.770627 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 16:01:37.771027 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.771012 2580 manager.go:349] Starting recovery of all containers Aug 04 16:01:37.779544 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.779530 2580 manager.go:354] Recovery completed Aug 04 16:01:37.779911 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:37.779887 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-42.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 16:01:37.783100 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.783089 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 16:01:37.785018 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.784996 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasSufficientMemory" Aug 04 16:01:37.785131 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.785028 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 16:01:37.785131 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.785039 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasSufficientPID" Aug 04 16:01:37.785921 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.785903 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 16:01:37.785921 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.785921 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 16:01:37.786034 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.785942 2580 state_mem.go:36] "Initialized new in-memory state store" Aug 04 16:01:37.786034 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.785964 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 16:01:37.786976 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.786960 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasSufficientMemory" Aug 04 16:01:37.787049 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.786985 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 16:01:37.787049 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.786995 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasSufficientPID" Aug 04 16:01:37.787430 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.787409 2580 policy_none.go:49] "None policy: Start" Aug 04 16:01:37.787473 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.787443 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 16:01:37.787473 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.787458 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 16:01:37.788460 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.788449 2580 policy_none.go:47] "Start" Aug 04 16:01:37.796012 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:37.795873 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-42.ec2.internal.18c8a591c74df997 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-42.ec2.internal,UID:ip-10-0-132-42.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-42.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-42.ec2.internal,},FirstTimestamp:2026-08-04 16:01:37.785018775 +0000 UTC m=+0.332414201,LastTimestamp:2026-08-04 16:01:37.785018775 +0000 UTC m=+0.332414201,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-42.ec2.internal,}" Aug 04 16:01:37.799253 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.799238 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d4xgz" Aug 04 16:01:37.806462 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:37.806389 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-42.ec2.internal.18c8a591c74e3344 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-42.ec2.internal,UID:ip-10-0-132-42.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-42.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-42.ec2.internal,},FirstTimestamp:2026-08-04 16:01:37.78503354 +0000 UTC m=+0.332428964,LastTimestamp:2026-08-04 16:01:37.78503354 +0000 UTC m=+0.332428964,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-42.ec2.internal,}" Aug 04 16:01:37.810951 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.810930 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d4xgz" Aug 04 16:01:37.816363 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:37.816303 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-42.ec2.internal.18c8a591c74e5aa5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-42.ec2.internal,UID:ip-10-0-132-42.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-42.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-42.ec2.internal,},FirstTimestamp:2026-08-04 16:01:37.785043621 +0000 UTC m=+0.332439045,LastTimestamp:2026-08-04 16:01:37.785043621 +0000 UTC m=+0.332439045,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-42.ec2.internal,}" Aug 04 16:01:37.826781 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.826762 2580 manager.go:337] "Starting Device Plugin manager" Aug 04 16:01:37.826869 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:37.826786 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 16:01:37.826869 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.826796 2580 server.go:87] "Starting device plugin registration server" Aug 04 16:01:37.826999 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.826988 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 16:01:37.827043 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.827000 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 16:01:37.827097 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.827072 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 16:01:37.827156 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.827146 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 16:01:37.827208 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.827158 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 16:01:37.827558 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:37.827537 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 16:01:37.827644 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:37.827591 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-42.ec2.internal\" not found" Aug 04 16:01:37.908482 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.908459 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 16:01:37.909721 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.909562 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 16:01:37.909721 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.909591 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 16:01:37.909721 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.909607 2580 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 16:01:37.909721 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:37.909633 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 16:01:37.911845 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.911830 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 16:01:37.927429 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.927412 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 16:01:37.928083 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.928069 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasSufficientMemory" Aug 04 16:01:37.928149 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.928092 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 16:01:37.928149 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.928103 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasSufficientPID" Aug 04 16:01:37.928149 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.928121 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-42.ec2.internal" Aug 04 16:01:37.937000 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:37.936988 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-42.ec2.internal" Aug 04 16:01:37.937042 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:37.937005 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-42.ec2.internal\": node \"ip-10-0-132-42.ec2.internal\" not found" Aug 04 16:01:37.962618 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:37.962578 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-42.ec2.internal\" not found" Aug 04 16:01:38.010562 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.010545 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-42.ec2.internal"] Aug 04 16:01:38.010630 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.010594 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 16:01:38.011206 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.011192 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasSufficientMemory" Aug 04 16:01:38.011272 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.011231 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 16:01:38.011272 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.011247 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasSufficientPID" Aug 04 16:01:38.012567 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.012556 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 16:01:38.012680 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.012669 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal" Aug 04 16:01:38.012713 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.012694 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 16:01:38.013197 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.013179 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasSufficientMemory" Aug 04 16:01:38.013295 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.013204 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 16:01:38.013295 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.013229 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasSufficientPID" Aug 04 16:01:38.013799 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.013783 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasSufficientMemory" Aug 04 16:01:38.013874 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.013809 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 16:01:38.013874 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.013819 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasSufficientPID" Aug 04 16:01:38.015049 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.015029 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-42.ec2.internal" Aug 04 16:01:38.015104 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.015059 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 16:01:38.015630 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.015615 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasSufficientMemory" Aug 04 16:01:38.015705 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.015642 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 16:01:38.015705 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.015655 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeHasSufficientPID" Aug 04 16:01:38.030725 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:38.030708 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-42.ec2.internal\" not found" node="ip-10-0-132-42.ec2.internal" Aug 04 16:01:38.034945 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:38.034930 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-42.ec2.internal\" not found" node="ip-10-0-132-42.ec2.internal" Aug 04 16:01:38.063525 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:38.063507 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-42.ec2.internal\" not found" Aug 04 16:01:38.071246 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.071214 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8a12c8c42b46b94e7355712b1c24093-config\") pod \"kube-apiserver-proxy-ip-10-0-132-42.ec2.internal\" (UID: \"c8a12c8c42b46b94e7355712b1c24093\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-42.ec2.internal" Aug 04 16:01:38.071332 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.071249 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7188073945aff2aff7853ad3566ec4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal\" (UID: \"b7188073945aff2aff7853ad3566ec4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal" Aug 04 16:01:38.071332 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.071266 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7188073945aff2aff7853ad3566ec4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal\" (UID: \"b7188073945aff2aff7853ad3566ec4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal" Aug 04 16:01:38.164156 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:38.164132 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-42.ec2.internal\" not found" Aug 04 16:01:38.171397 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.171379 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8a12c8c42b46b94e7355712b1c24093-config\") pod \"kube-apiserver-proxy-ip-10-0-132-42.ec2.internal\" (UID: \"c8a12c8c42b46b94e7355712b1c24093\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-42.ec2.internal" Aug 04 16:01:38.171457 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.171402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7188073945aff2aff7853ad3566ec4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal\" (UID: \"b7188073945aff2aff7853ad3566ec4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal" Aug 04 16:01:38.171457 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.171429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7188073945aff2aff7853ad3566ec4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal\" (UID: \"b7188073945aff2aff7853ad3566ec4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal" Aug 04 16:01:38.171546 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.171460 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7188073945aff2aff7853ad3566ec4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal\" (UID: \"b7188073945aff2aff7853ad3566ec4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal" Aug 04 16:01:38.171546 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.171464 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7188073945aff2aff7853ad3566ec4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal\" (UID: \"b7188073945aff2aff7853ad3566ec4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal" Aug 04 16:01:38.171546 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:38.171488 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8a12c8c42b46b94e7355712b1c24093-config\") pod \"kube-apiserver-proxy-ip-10-0-132-42.ec2.internal\" (UID: \"c8a12c8c42b46b94e7355712b1c24093\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-42.ec2.internal" Aug 04 16:01:39.667897 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:39.667853 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-42.ec2.internal\" not found" Aug 04 16:01:39.736304 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:39.736273 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal" Aug 04 16:01:39.741307 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:39.741274 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-42.ec2.internal" Aug 04 16:01:39.768200 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:39.768179 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-42.ec2.internal\" not found" Aug 04 16:01:39.868270 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:39.868245 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-42.ec2.internal\" not found" Aug 04 16:01:39.968769 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:39.968722 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-42.ec2.internal\" not found" Aug 04 16:01:40.069414 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.069391 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-42.ec2.internal\" not found" Aug 04 16:01:40.114903 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.114887 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 16:01:40.115259 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.115027 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 16:01:40.139337 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.139316 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 16:01:40.152584 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.152569 2580 apiserver.go:52] "Watching apiserver" Aug 04 16:01:40.163703 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.163683 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 16:01:40.165257 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.165237 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-8sdq7","openshift-multus/network-metrics-daemon-t79jc","openshift-ovn-kubernetes/ovnkube-node-gjgb9","kube-system/konnectivity-agent-lnn4g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97","openshift-dns/node-resolver-g2845","openshift-image-registry/node-ca-vbhgz","openshift-network-diagnostics/network-check-target-qvmnx","openshift-network-operator/iptables-alerter-nppcn","kube-system/global-pull-secret-syncer-5hhk7","openshift-cluster-node-tuning-operator/tuned-kknn8","openshift-multus/multus-7q9g5"] Aug 04 16:01:40.166959 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.166939 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.168132 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.168113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:40.168216 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.168183 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79jc" podUID="56a1a012-b25b-4c19-b6be-73f382416c22" Aug 04 16:01:40.169490 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.169469 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.170073 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.170056 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 16:01:40.170266 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.170250 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 16:01:40.170364 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.170348 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 16:01:40.170418 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.170399 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 16:01:40.170486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.170451 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 16:01:40.170548 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.170532 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-snrc2\"" Aug 04 16:01:40.170714 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.170699 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lnn4g" Aug 04 16:01:40.171504 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.171490 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 16:01:40.172051 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.172036 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 16:01:40.172315 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.172295 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.172315 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.172313 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal" Aug 04 16:01:40.172464 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.172330 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 16:01:40.172564 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.172510 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 16:01:40.172685 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.172649 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 16:01:40.172948 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.172933 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m9xcv\"" Aug 04 16:01:40.173096 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.173084 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 16:01:40.173265 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.173247 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 16:01:40.173512 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.173495 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 16:01:40.173595 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.173499 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 16:01:40.173748 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.173735 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6z985\"" Aug 04 16:01:40.173974 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.173961 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g2845" Aug 04 16:01:40.174661 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.174646 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zt7ql\"" Aug 04 16:01:40.175110 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.175094 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 16:01:40.175192 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.175121 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 16:01:40.175192 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.175146 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 16:01:40.175359 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.175342 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vbhgz" Aug 04 16:01:40.176549 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.176534 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 16:01:40.176888 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.176871 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 16:01:40.176968 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.176874 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-422bp\"" Aug 04 16:01:40.177023 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.176990 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:40.177063 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.177038 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvmnx" podUID="fc84cbe7-6fbf-4152-9bcb-6227c4126473" Aug 04 16:01:40.177651 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.177636 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sbjbh\"" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178103 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-run-systemd\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178191 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/656bb7e0-4acf-401f-a87a-209fa05614fa-agent-certs\") pod \"konnectivity-agent-lnn4g\" (UID: \"656bb7e0-4acf-401f-a87a-209fa05614fa\") " pod="kube-system/konnectivity-agent-lnn4g" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178229 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/656bb7e0-4acf-401f-a87a-209fa05614fa-konnectivity-ca\") pod \"konnectivity-agent-lnn4g\" (UID: \"656bb7e0-4acf-401f-a87a-209fa05614fa\") " pod="kube-system/konnectivity-agent-lnn4g" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178320 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178351 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a9c0f306-ee82-4a4a-9680-5b0835964025-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178386 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a9c0f306-ee82-4a4a-9680-5b0835964025-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178463 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkkjf\" (UniqueName: \"kubernetes.io/projected/a9c0f306-ee82-4a4a-9680-5b0835964025-kube-api-access-qkkjf\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178510 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-cni-bin\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0710ce46-7acc-47cf-8182-b1248d751567-ovnkube-config\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178560 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0710ce46-7acc-47cf-8182-b1248d751567-ovn-node-metrics-cert\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178653 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-node-log\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178678 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178680 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-socket-dir\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.179778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178711 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-etc-selinux\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.180614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178738 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add717dd-d023-43b7-bf8b-f50516f053db-tmp-dir\") pod \"node-resolver-g2845\" (UID: \"add717dd-d023-43b7-bf8b-f50516f053db\") " pod="openshift-dns/node-resolver-g2845" Aug 04 16:01:40.180614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9c0f306-ee82-4a4a-9680-5b0835964025-system-cni-dir\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.180614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178790 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-run-netns\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.180614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178817 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-log-socket\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.180614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-run-ovn-kubernetes\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.180614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178902 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-cni-netd\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.180614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqq6h\" (UniqueName: \"kubernetes.io/projected/add717dd-d023-43b7-bf8b-f50516f053db-kube-api-access-jqq6h\") pod \"node-resolver-g2845\" (UID: \"add717dd-d023-43b7-bf8b-f50516f053db\") " pod="openshift-dns/node-resolver-g2845" Aug 04 16:01:40.180614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178952 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5jpn\" (UniqueName: \"kubernetes.io/projected/56a1a012-b25b-4c19-b6be-73f382416c22-kube-api-access-f5jpn\") pod \"network-metrics-daemon-t79jc\" (UID: \"56a1a012-b25b-4c19-b6be-73f382416c22\") " pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:40.180614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.178980 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-slash\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.180614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179016 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-var-lib-openvswitch\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.180614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179040 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-etc-openvswitch\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.180614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179115 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0710ce46-7acc-47cf-8182-b1248d751567-ovnkube-script-lib\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.180614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179150 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-sys-fs\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.180614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0710ce46-7acc-47cf-8182-b1248d751567-env-overrides\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.180614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179205 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xr6g\" (UniqueName: \"kubernetes.io/projected/1b623ab2-5a3b-47a7-accb-ce513e36b597-kube-api-access-7xr6g\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.180614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179232 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-kubelet\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.181294 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179260 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-registration-dir\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.181294 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179310 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-device-dir\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.181294 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/add717dd-d023-43b7-bf8b-f50516f053db-hosts-file\") pod \"node-resolver-g2845\" (UID: \"add717dd-d023-43b7-bf8b-f50516f053db\") " pod="openshift-dns/node-resolver-g2845" Aug 04 16:01:40.181294 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179364 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9c0f306-ee82-4a4a-9680-5b0835964025-cnibin\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.181294 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179393 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-systemd-units\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.181294 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-run-openvswitch\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.181294 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179452 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-run-ovn\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.181294 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179481 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc5h2\" (UniqueName: \"kubernetes.io/projected/0710ce46-7acc-47cf-8182-b1248d751567-kube-api-access-qc5h2\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.181294 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a9c0f306-ee82-4a4a-9680-5b0835964025-os-release\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.181294 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9c0f306-ee82-4a4a-9680-5b0835964025-cni-binary-copy\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.181294 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179561 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9c0f306-ee82-4a4a-9680-5b0835964025-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.181294 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs\") pod \"network-metrics-daemon-t79jc\" (UID: \"56a1a012-b25b-4c19-b6be-73f382416c22\") " pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:40.181294 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.179854 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nppcn" Aug 04 16:01:40.181294 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.181142 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:40.181294 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.181233 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:01:40.182362 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.182342 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 16:01:40.182575 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.182558 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tflwv\"" Aug 04 16:01:40.182877 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.182858 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 16:01:40.182972 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.182909 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 16:01:40.182972 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.182961 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.184131 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.184115 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.185798 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.185772 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 16:01:40.185798 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.185778 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 16:01:40.185952 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.185831 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b58vm\"" Aug 04 16:01:40.187002 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.186969 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fbpww\"" Aug 04 16:01:40.187098 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.187081 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 16:01:40.187203 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.187085 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 16:01:40.189564 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.189548 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 16:01:40.189654 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.189622 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-42.ec2.internal" Aug 04 16:01:40.189785 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.189763 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal"] Aug 04 16:01:40.198686 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.198670 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-42.ec2.internal"] Aug 04 16:01:40.198765 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.198754 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 16:01:40.214580 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.214565 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h97zr" Aug 04 16:01:40.218965 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.218900 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 15:56:39 +0000 UTC" deadline="2028-01-05 00:57:20.275158138 +0000 UTC" Aug 04 16:01:40.218965 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.218939 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12440h55m40.056220752s" Aug 04 16:01:40.221163 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.221146 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h97zr" Aug 04 16:01:40.273792 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.273771 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 16:01:40.280694 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.280673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-multus-conf-dir\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.280781 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.280704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-etc-kubernetes\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.280781 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.280719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29da60cb-4da1-44b2-a158-3532bf7c5c0d-iptables-alerter-script\") pod \"iptables-alerter-nppcn\" (UID: \"29da60cb-4da1-44b2-a158-3532bf7c5c0d\") " pod="openshift-network-operator/iptables-alerter-nppcn" Aug 04 16:01:40.280781 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.280740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a9c0f306-ee82-4a4a-9680-5b0835964025-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.280882 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.280778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a9c0f306-ee82-4a4a-9680-5b0835964025-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.280941 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.280927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkkjf\" (UniqueName: \"kubernetes.io/projected/a9c0f306-ee82-4a4a-9680-5b0835964025-kube-api-access-qkkjf\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.281011 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.280996 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0710ce46-7acc-47cf-8182-b1248d751567-ovnkube-config\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.281067 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281025 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0710ce46-7acc-47cf-8182-b1248d751567-ovn-node-metrics-cert\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.281067 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281056 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-kubelet-config\") pod \"global-pull-secret-syncer-5hhk7\" (UID: \"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db\") " pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:40.281160 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281081 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-node-log\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.281160 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281111 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-kubernetes\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.281160 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281135 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-multus-socket-dir-parent\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.281313 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-hostroot\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.281313 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281160 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-node-log\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.281313 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-socket-dir\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.281313 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-etc-selinux\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.281313 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281218 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a9c0f306-ee82-4a4a-9680-5b0835964025-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.281313 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add717dd-d023-43b7-bf8b-f50516f053db-tmp-dir\") pod \"node-resolver-g2845\" (UID: \"add717dd-d023-43b7-bf8b-f50516f053db\") " pod="openshift-dns/node-resolver-g2845" Aug 04 16:01:40.281313 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9c0f306-ee82-4a4a-9680-5b0835964025-system-cni-dir\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.281313 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281255 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a9c0f306-ee82-4a4a-9680-5b0835964025-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.281313 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-log-socket\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.281627 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281338 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-log-socket\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.281627 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281320 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9c0f306-ee82-4a4a-9680-5b0835964025-system-cni-dir\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.281793 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281299 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-socket-dir\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.281793 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281674 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-etc-selinux\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.281892 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281797 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-cni-netd\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.281892 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281806 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 16:01:40.281989 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281930 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add717dd-d023-43b7-bf8b-f50516f053db-tmp-dir\") pod \"node-resolver-g2845\" (UID: \"add717dd-d023-43b7-bf8b-f50516f053db\") " pod="openshift-dns/node-resolver-g2845" Aug 04 16:01:40.282067 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.281799 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-cni-netd\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.282119 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-lib-modules\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.282168 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-multus-cni-dir\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.282217 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqq6h\" (UniqueName: \"kubernetes.io/projected/add717dd-d023-43b7-bf8b-f50516f053db-kube-api-access-jqq6h\") pod \"node-resolver-g2845\" (UID: \"add717dd-d023-43b7-bf8b-f50516f053db\") " pod="openshift-dns/node-resolver-g2845" Aug 04 16:01:40.282217 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282199 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-slash\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.282332 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282230 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-sysconfig\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.282332 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56b8e09d-90bb-401d-96a6-20d691168fc3-host\") pod \"node-ca-vbhgz\" (UID: \"56b8e09d-90bb-401d-96a6-20d691168fc3\") " pod="openshift-image-registry/node-ca-vbhgz" Aug 04 16:01:40.282421 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0710ce46-7acc-47cf-8182-b1248d751567-ovnkube-config\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.282421 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282337 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45b6b\" (UniqueName: \"kubernetes.io/projected/29da60cb-4da1-44b2-a158-3532bf7c5c0d-kube-api-access-45b6b\") pod \"iptables-alerter-nppcn\" (UID: \"29da60cb-4da1-44b2-a158-3532bf7c5c0d\") " pod="openshift-network-operator/iptables-alerter-nppcn" Aug 04 16:01:40.282421 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0710ce46-7acc-47cf-8182-b1248d751567-env-overrides\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.282421 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282403 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-slash\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.282421 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282409 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-cnibin\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.282648 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282517 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-run-k8s-cni-cncf-io\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.282648 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282583 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-var-lib-cni-bin\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.282648 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282631 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4dlq\" (UniqueName: \"kubernetes.io/projected/56b8e09d-90bb-401d-96a6-20d691168fc3-kube-api-access-q4dlq\") pod \"node-ca-vbhgz\" (UID: \"56b8e09d-90bb-401d-96a6-20d691168fc3\") " pod="openshift-image-registry/node-ca-vbhgz" Aug 04 16:01:40.282763 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xr6g\" (UniqueName: \"kubernetes.io/projected/1b623ab2-5a3b-47a7-accb-ce513e36b597-kube-api-access-7xr6g\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.282763 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282687 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-systemd\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.282763 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-tuned\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.282900 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282841 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-run-multus-certs\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.282900 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-registration-dir\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.283012 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282902 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0710ce46-7acc-47cf-8182-b1248d751567-env-overrides\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.283012 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-run-openvswitch\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.283012 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-run-ovn\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.283012 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282972 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/656bb7e0-4acf-401f-a87a-209fa05614fa-agent-certs\") pod \"konnectivity-agent-lnn4g\" (UID: \"656bb7e0-4acf-401f-a87a-209fa05614fa\") " pod="kube-system/konnectivity-agent-lnn4g" Aug 04 16:01:40.283012 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.282995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-sysctl-d\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.283259 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283022 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-run-netns\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.283259 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-multus-daemon-config\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.283259 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283079 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a9c0f306-ee82-4a4a-9680-5b0835964025-os-release\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.283259 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9c0f306-ee82-4a4a-9680-5b0835964025-cni-binary-copy\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.283259 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9c0f306-ee82-4a4a-9680-5b0835964025-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.283259 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283175 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs\") pod \"network-metrics-daemon-t79jc\" (UID: \"56a1a012-b25b-4c19-b6be-73f382416c22\") " pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:40.283259 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283204 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-var-lib-kubelet\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.283259 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283231 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29da60cb-4da1-44b2-a158-3532bf7c5c0d-host-slash\") pod \"iptables-alerter-nppcn\" (UID: \"29da60cb-4da1-44b2-a158-3532bf7c5c0d\") " pod="openshift-network-operator/iptables-alerter-nppcn" Aug 04 16:01:40.283259 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283254 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.283682 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-cni-bin\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.283682 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283325 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-var-lib-kubelet\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.283682 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-system-cni-dir\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.283682 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283384 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-cni-binary-copy\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.283682 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283409 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-sysctl-conf\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.283682 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283423 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-registration-dir\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.283682 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283501 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-run-ovn\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.283682 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283535 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.283682 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283579 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-cni-bin\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.283682 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.283596 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 16:01:40.283682 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.283672 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs podName:56a1a012-b25b-4c19-b6be-73f382416c22 nodeName:}" failed. No retries permitted until 2026-08-04 16:01:40.783646004 +0000 UTC m=+1.927978241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs") pod "network-metrics-daemon-t79jc" (UID: "56a1a012-b25b-4c19-b6be-73f382416c22") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 16:01:40.284192 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283850 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9c0f306-ee82-4a4a-9680-5b0835964025-cni-binary-copy\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.284192 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283931 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-run-openvswitch\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.284192 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.283971 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-sys\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.284192 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.284008 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-run-netns\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.284192 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.284038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-run-ovn-kubernetes\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.284192 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.284059 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9c0f306-ee82-4a4a-9680-5b0835964025-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.284192 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.284064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0710ce46-7acc-47cf-8182-b1248d751567-ovnkube-script-lib\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.284192 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.284142 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a9c0f306-ee82-4a4a-9680-5b0835964025-os-release\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.284192 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.284190 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-run-netns\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.284634 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.284235 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-run-ovn-kubernetes\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.284634 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.284528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0710ce46-7acc-47cf-8182-b1248d751567-ovnkube-script-lib\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.285885 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.285671 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0710ce46-7acc-47cf-8182-b1248d751567-ovn-node-metrics-cert\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.285885 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.285702 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jpn\" (UniqueName: \"kubernetes.io/projected/56a1a012-b25b-4c19-b6be-73f382416c22-kube-api-access-f5jpn\") pod \"network-metrics-daemon-t79jc\" (UID: \"56a1a012-b25b-4c19-b6be-73f382416c22\") " pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:40.285885 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.285775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-var-lib-openvswitch\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.285885 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.285810 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-etc-openvswitch\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.285885 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.285848 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-modprobe-d\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.286163 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.285928 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-etc-openvswitch\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.286163 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.285946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-run\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.286163 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.285931 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-var-lib-openvswitch\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.286163 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.285999 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k8mq\" (UniqueName: \"kubernetes.io/projected/dba3d3f9-effd-4c57-a584-6a79d1fc4516-kube-api-access-8k8mq\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.286163 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret\") pod \"global-pull-secret-syncer-5hhk7\" (UID: \"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db\") " pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:40.286163 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-sys-fs\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.286163 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286102 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-host\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.286163 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286121 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-var-lib-cni-multus\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.286494 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286167 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-sys-fs\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.286494 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286456 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-kubelet\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.286561 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286491 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-kubelet\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.286561 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxhj2\" (UniqueName: \"kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2\") pod \"network-check-target-qvmnx\" (UID: \"fc84cbe7-6fbf-4152-9bcb-6227c4126473\") " pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:40.286561 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286530 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dba3d3f9-effd-4c57-a584-6a79d1fc4516-tmp\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.286561 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj45q\" (UniqueName: \"kubernetes.io/projected/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-kube-api-access-qj45q\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.286732 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-device-dir\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.286732 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/add717dd-d023-43b7-bf8b-f50516f053db-hosts-file\") pod \"node-resolver-g2845\" (UID: \"add717dd-d023-43b7-bf8b-f50516f053db\") " pod="openshift-dns/node-resolver-g2845" Aug 04 16:01:40.286732 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9c0f306-ee82-4a4a-9680-5b0835964025-cnibin\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.286732 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-systemd-units\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.286732 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc5h2\" (UniqueName: \"kubernetes.io/projected/0710ce46-7acc-47cf-8182-b1248d751567-kube-api-access-qc5h2\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.286732 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286648 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/656bb7e0-4acf-401f-a87a-209fa05614fa-konnectivity-ca\") pod \"konnectivity-agent-lnn4g\" (UID: \"656bb7e0-4acf-401f-a87a-209fa05614fa\") " pod="kube-system/konnectivity-agent-lnn4g" Aug 04 16:01:40.286732 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286656 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/656bb7e0-4acf-401f-a87a-209fa05614fa-agent-certs\") pod \"konnectivity-agent-lnn4g\" (UID: \"656bb7e0-4acf-401f-a87a-209fa05614fa\") " pod="kube-system/konnectivity-agent-lnn4g" Aug 04 16:01:40.286732 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286648 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1b623ab2-5a3b-47a7-accb-ce513e36b597-device-dir\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.286732 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286671 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9c0f306-ee82-4a4a-9680-5b0835964025-cnibin\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.286732 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286688 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-systemd-units\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.286732 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286709 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56b8e09d-90bb-401d-96a6-20d691168fc3-serviceca\") pod \"node-ca-vbhgz\" (UID: \"56b8e09d-90bb-401d-96a6-20d691168fc3\") " pod="openshift-image-registry/node-ca-vbhgz" Aug 04 16:01:40.286732 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286714 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/add717dd-d023-43b7-bf8b-f50516f053db-hosts-file\") pod \"node-resolver-g2845\" (UID: \"add717dd-d023-43b7-bf8b-f50516f053db\") " pod="openshift-dns/node-resolver-g2845" Aug 04 16:01:40.286732 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-dbus\") pod \"global-pull-secret-syncer-5hhk7\" (UID: \"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db\") " pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:40.287190 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286763 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-run-systemd\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.287190 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286788 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.287190 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286798 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-run-systemd\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.287190 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-os-release\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.287190 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.286859 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0710ce46-7acc-47cf-8182-b1248d751567-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.287190 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.287033 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/656bb7e0-4acf-401f-a87a-209fa05614fa-konnectivity-ca\") pod \"konnectivity-agent-lnn4g\" (UID: \"656bb7e0-4acf-401f-a87a-209fa05614fa\") " pod="kube-system/konnectivity-agent-lnn4g" Aug 04 16:01:40.293156 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.293137 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkkjf\" (UniqueName: \"kubernetes.io/projected/a9c0f306-ee82-4a4a-9680-5b0835964025-kube-api-access-qkkjf\") pod \"multus-additional-cni-plugins-8sdq7\" (UID: \"a9c0f306-ee82-4a4a-9680-5b0835964025\") " pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.296508 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.296482 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xr6g\" (UniqueName: \"kubernetes.io/projected/1b623ab2-5a3b-47a7-accb-ce513e36b597-kube-api-access-7xr6g\") pod \"aws-ebs-csi-driver-node-ljb97\" (UID: \"1b623ab2-5a3b-47a7-accb-ce513e36b597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.296704 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.296690 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5jpn\" (UniqueName: \"kubernetes.io/projected/56a1a012-b25b-4c19-b6be-73f382416c22-kube-api-access-f5jpn\") pod \"network-metrics-daemon-t79jc\" (UID: \"56a1a012-b25b-4c19-b6be-73f382416c22\") " pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:40.297373 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.297354 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc5h2\" (UniqueName: \"kubernetes.io/projected/0710ce46-7acc-47cf-8182-b1248d751567-kube-api-access-qc5h2\") pod \"ovnkube-node-gjgb9\" (UID: \"0710ce46-7acc-47cf-8182-b1248d751567\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.297439 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.297400 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqq6h\" (UniqueName: \"kubernetes.io/projected/add717dd-d023-43b7-bf8b-f50516f053db-kube-api-access-jqq6h\") pod \"node-resolver-g2845\" (UID: \"add717dd-d023-43b7-bf8b-f50516f053db\") " pod="openshift-dns/node-resolver-g2845" Aug 04 16:01:40.312225 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:40.312203 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7188073945aff2aff7853ad3566ec4f.slice/crio-c3334c0565733621e749eb59c352e9cc7bee3a08e3a3e9b26b5dcb706b3a5c9f WatchSource:0}: Error finding container c3334c0565733621e749eb59c352e9cc7bee3a08e3a3e9b26b5dcb706b3a5c9f: Status 404 returned error can't find the container with id c3334c0565733621e749eb59c352e9cc7bee3a08e3a3e9b26b5dcb706b3a5c9f Aug 04 16:01:40.312542 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:40.312522 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a12c8c42b46b94e7355712b1c24093.slice/crio-6d86e1d69e034edd93c16c401f0849c79e17c6cd3873f694fa1aa017e450ce00 WatchSource:0}: Error finding container 6d86e1d69e034edd93c16c401f0849c79e17c6cd3873f694fa1aa017e450ce00: Status 404 returned error can't find the container with id 6d86e1d69e034edd93c16c401f0849c79e17c6cd3873f694fa1aa017e450ce00 Aug 04 16:01:40.316609 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.316596 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:01:40.387523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387501 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhj2\" (UniqueName: \"kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2\") pod \"network-check-target-qvmnx\" (UID: \"fc84cbe7-6fbf-4152-9bcb-6227c4126473\") " pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:40.387618 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dba3d3f9-effd-4c57-a584-6a79d1fc4516-tmp\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.387618 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387543 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj45q\" (UniqueName: \"kubernetes.io/projected/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-kube-api-access-qj45q\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.387618 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387558 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56b8e09d-90bb-401d-96a6-20d691168fc3-serviceca\") pod \"node-ca-vbhgz\" (UID: \"56b8e09d-90bb-401d-96a6-20d691168fc3\") " pod="openshift-image-registry/node-ca-vbhgz" Aug 04 16:01:40.387618 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-dbus\") pod \"global-pull-secret-syncer-5hhk7\" (UID: \"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db\") " pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:40.387814 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-os-release\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.387814 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-multus-conf-dir\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.387814 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-etc-kubernetes\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.387814 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29da60cb-4da1-44b2-a158-3532bf7c5c0d-iptables-alerter-script\") pod \"iptables-alerter-nppcn\" (UID: \"29da60cb-4da1-44b2-a158-3532bf7c5c0d\") " pod="openshift-network-operator/iptables-alerter-nppcn" Aug 04 16:01:40.387814 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387806 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-etc-kubernetes\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.388040 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387815 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-multus-conf-dir\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.388040 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387840 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-os-release\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.388040 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-kubelet-config\") pod \"global-pull-secret-syncer-5hhk7\" (UID: \"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db\") " pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:40.388040 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387894 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-kubernetes\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.388040 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387919 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-dbus\") pod \"global-pull-secret-syncer-5hhk7\" (UID: \"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db\") " pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:40.388040 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-multus-socket-dir-parent\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.388040 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387938 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56b8e09d-90bb-401d-96a6-20d691168fc3-serviceca\") pod \"node-ca-vbhgz\" (UID: \"56b8e09d-90bb-401d-96a6-20d691168fc3\") " pod="openshift-image-registry/node-ca-vbhgz" Aug 04 16:01:40.388040 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387963 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-hostroot\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.388040 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387970 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-kubelet-config\") pod \"global-pull-secret-syncer-5hhk7\" (UID: \"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db\") " pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:40.388040 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387979 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-kubernetes\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.388040 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.387992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-lib-modules\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.388040 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388010 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-multus-socket-dir-parent\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.388040 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388014 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-multus-cni-dir\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.388040 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388022 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-hostroot\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.388040 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388041 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-sysconfig\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56b8e09d-90bb-401d-96a6-20d691168fc3-host\") pod \"node-ca-vbhgz\" (UID: \"56b8e09d-90bb-401d-96a6-20d691168fc3\") " pod="openshift-image-registry/node-ca-vbhgz" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388078 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-multus-cni-dir\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45b6b\" (UniqueName: \"kubernetes.io/projected/29da60cb-4da1-44b2-a158-3532bf7c5c0d-kube-api-access-45b6b\") pod \"iptables-alerter-nppcn\" (UID: \"29da60cb-4da1-44b2-a158-3532bf7c5c0d\") " pod="openshift-network-operator/iptables-alerter-nppcn" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388100 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-sysconfig\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388136 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56b8e09d-90bb-401d-96a6-20d691168fc3-host\") pod \"node-ca-vbhgz\" (UID: \"56b8e09d-90bb-401d-96a6-20d691168fc3\") " pod="openshift-image-registry/node-ca-vbhgz" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-cnibin\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-run-k8s-cni-cncf-io\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388221 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-var-lib-cni-bin\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388239 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29da60cb-4da1-44b2-a158-3532bf7c5c0d-iptables-alerter-script\") pod \"iptables-alerter-nppcn\" (UID: \"29da60cb-4da1-44b2-a158-3532bf7c5c0d\") " pod="openshift-network-operator/iptables-alerter-nppcn" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388245 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4dlq\" (UniqueName: \"kubernetes.io/projected/56b8e09d-90bb-401d-96a6-20d691168fc3-kube-api-access-q4dlq\") pod \"node-ca-vbhgz\" (UID: \"56b8e09d-90bb-401d-96a6-20d691168fc3\") " pod="openshift-image-registry/node-ca-vbhgz" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388101 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-lib-modules\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-systemd\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388302 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-run-k8s-cni-cncf-io\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-tuned\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388296 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-cnibin\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388334 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-var-lib-cni-bin\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-run-multus-certs\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.388695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388324 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-systemd\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-sysctl-d\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388421 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-run-multus-certs\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388444 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-sysctl-d\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-run-netns\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388481 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-multus-daemon-config\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388479 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-run-netns\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-var-lib-kubelet\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29da60cb-4da1-44b2-a158-3532bf7c5c0d-host-slash\") pod \"iptables-alerter-nppcn\" (UID: \"29da60cb-4da1-44b2-a158-3532bf7c5c0d\") " pod="openshift-network-operator/iptables-alerter-nppcn" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-var-lib-kubelet\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-system-cni-dir\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388638 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-var-lib-kubelet\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388648 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-cni-binary-copy\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388665 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29da60cb-4da1-44b2-a158-3532bf7c5c0d-host-slash\") pod \"iptables-alerter-nppcn\" (UID: \"29da60cb-4da1-44b2-a158-3532bf7c5c0d\") " pod="openshift-network-operator/iptables-alerter-nppcn" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388676 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-var-lib-kubelet\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-sysctl-conf\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388721 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-sys\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388735 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-system-cni-dir\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.389414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-modprobe-d\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.389895 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388769 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-run\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.389895 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388779 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-sys\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.389895 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k8mq\" (UniqueName: \"kubernetes.io/projected/dba3d3f9-effd-4c57-a584-6a79d1fc4516-kube-api-access-8k8mq\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.389895 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388809 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-sysctl-conf\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.389895 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret\") pod \"global-pull-secret-syncer-5hhk7\" (UID: \"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db\") " pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:40.389895 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-host\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.389895 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388850 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-modprobe-d\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.389895 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388859 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-run\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.389895 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-var-lib-cni-multus\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.389895 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388915 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-multus-daemon-config\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.389895 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.388932 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 16:01:40.389895 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.388989 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret podName:cb9271e1-f3f4-4afc-b1ce-22d79e60a3db nodeName:}" failed. No retries permitted until 2026-08-04 16:01:40.888970979 +0000 UTC m=+2.033303204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret") pod "global-pull-secret-syncer-5hhk7" (UID: "cb9271e1-f3f4-4afc-b1ce-22d79e60a3db") : object "kube-system"/"original-pull-secret" not registered Aug 04 16:01:40.389895 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.388920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dba3d3f9-effd-4c57-a584-6a79d1fc4516-host\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.389895 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.389039 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-host-var-lib-cni-multus\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.389895 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.389088 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-cni-binary-copy\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.389895 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.389645 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dba3d3f9-effd-4c57-a584-6a79d1fc4516-tmp\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.390338 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.390061 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dba3d3f9-effd-4c57-a584-6a79d1fc4516-etc-tuned\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.396360 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.396345 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45b6b\" (UniqueName: \"kubernetes.io/projected/29da60cb-4da1-44b2-a158-3532bf7c5c0d-kube-api-access-45b6b\") pod \"iptables-alerter-nppcn\" (UID: \"29da60cb-4da1-44b2-a158-3532bf7c5c0d\") " pod="openshift-network-operator/iptables-alerter-nppcn" Aug 04 16:01:40.396577 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.396550 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 16:01:40.396577 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.396569 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 16:01:40.396577 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.396580 2580 projected.go:196] Error preparing data for projected volume kube-api-access-qxhj2 for pod openshift-network-diagnostics/network-check-target-qvmnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 16:01:40.396750 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.396624 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2 podName:fc84cbe7-6fbf-4152-9bcb-6227c4126473 nodeName:}" failed. No retries permitted until 2026-08-04 16:01:40.896611906 +0000 UTC m=+2.040944152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qxhj2" (UniqueName: "kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2") pod "network-check-target-qvmnx" (UID: "fc84cbe7-6fbf-4152-9bcb-6227c4126473") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 16:01:40.398219 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.398198 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj45q\" (UniqueName: \"kubernetes.io/projected/14b99219-a0ce-4d7b-a976-aa710a8fa6bf-kube-api-access-qj45q\") pod \"multus-7q9g5\" (UID: \"14b99219-a0ce-4d7b-a976-aa710a8fa6bf\") " pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.398522 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.398504 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k8mq\" (UniqueName: \"kubernetes.io/projected/dba3d3f9-effd-4c57-a584-6a79d1fc4516-kube-api-access-8k8mq\") pod \"tuned-kknn8\" (UID: \"dba3d3f9-effd-4c57-a584-6a79d1fc4516\") " pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.398648 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.398629 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4dlq\" (UniqueName: \"kubernetes.io/projected/56b8e09d-90bb-401d-96a6-20d691168fc3-kube-api-access-q4dlq\") pod \"node-ca-vbhgz\" (UID: \"56b8e09d-90bb-401d-96a6-20d691168fc3\") " pod="openshift-image-registry/node-ca-vbhgz" Aug 04 16:01:40.437354 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.437336 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 16:01:40.472156 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.472114 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 16:01:40.488871 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.488852 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8sdq7" Aug 04 16:01:40.497175 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:40.497156 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c0f306_ee82_4a4a_9680_5b0835964025.slice/crio-cf60898594c8e69c99411b891ab378d98b52aceb4cebcf6949c703e11761b701 WatchSource:0}: Error finding container cf60898594c8e69c99411b891ab378d98b52aceb4cebcf6949c703e11761b701: Status 404 returned error can't find the container with id cf60898594c8e69c99411b891ab378d98b52aceb4cebcf6949c703e11761b701 Aug 04 16:01:40.508364 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.508348 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:01:40.514846 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:40.514828 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0710ce46_7acc_47cf_8182_b1248d751567.slice/crio-6a5db26e5174a86b2036fe8a8c55604cc4155f913221a6c08b9fcd45513a9b4d WatchSource:0}: Error finding container 6a5db26e5174a86b2036fe8a8c55604cc4155f913221a6c08b9fcd45513a9b4d: Status 404 returned error can't find the container with id 6a5db26e5174a86b2036fe8a8c55604cc4155f913221a6c08b9fcd45513a9b4d Aug 04 16:01:40.515416 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.515397 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lnn4g" Aug 04 16:01:40.521816 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:40.521798 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656bb7e0_4acf_401f_a87a_209fa05614fa.slice/crio-28884941569f4c3d94f63a88dbe1e5b2d52a062b072901d683890989cac8e527 WatchSource:0}: Error finding container 28884941569f4c3d94f63a88dbe1e5b2d52a062b072901d683890989cac8e527: Status 404 returned error can't find the container with id 28884941569f4c3d94f63a88dbe1e5b2d52a062b072901d683890989cac8e527 Aug 04 16:01:40.528494 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.528478 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" Aug 04 16:01:40.534567 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:40.534549 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b623ab2_5a3b_47a7_accb_ce513e36b597.slice/crio-3da94ad68ea6ea8e0427daa76d205182f29dcccfd4ff88ea8694668b8e79fa1f WatchSource:0}: Error finding container 3da94ad68ea6ea8e0427daa76d205182f29dcccfd4ff88ea8694668b8e79fa1f: Status 404 returned error can't find the container with id 3da94ad68ea6ea8e0427daa76d205182f29dcccfd4ff88ea8694668b8e79fa1f Aug 04 16:01:40.543625 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.543609 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g2845" Aug 04 16:01:40.551771 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:40.551754 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd717dd_d023_43b7_bf8b_f50516f053db.slice/crio-9c7684c651be4f8588fa545563aad29b40178e90b1d1925a8adf3b8673b27ba8 WatchSource:0}: Error finding container 9c7684c651be4f8588fa545563aad29b40178e90b1d1925a8adf3b8673b27ba8: Status 404 returned error can't find the container with id 9c7684c651be4f8588fa545563aad29b40178e90b1d1925a8adf3b8673b27ba8 Aug 04 16:01:40.556445 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.556427 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vbhgz" Aug 04 16:01:40.562477 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:40.562456 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b8e09d_90bb_401d_96a6_20d691168fc3.slice/crio-6bc37e9713318263c6f3b7dd2a3bcae932165bc80dddfe4010b6bf3902022c7e WatchSource:0}: Error finding container 6bc37e9713318263c6f3b7dd2a3bcae932165bc80dddfe4010b6bf3902022c7e: Status 404 returned error can't find the container with id 6bc37e9713318263c6f3b7dd2a3bcae932165bc80dddfe4010b6bf3902022c7e Aug 04 16:01:40.563399 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.563380 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nppcn" Aug 04 16:01:40.571553 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:40.571533 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29da60cb_4da1_44b2_a158_3532bf7c5c0d.slice/crio-9eb6319e34f969b7e99eda9f46dd7b3742343f4843af4ee2b4605a497dd0cc7a WatchSource:0}: Error finding container 9eb6319e34f969b7e99eda9f46dd7b3742343f4843af4ee2b4605a497dd0cc7a: Status 404 returned error can't find the container with id 9eb6319e34f969b7e99eda9f46dd7b3742343f4843af4ee2b4605a497dd0cc7a Aug 04 16:01:40.573055 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.573040 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kknn8" Aug 04 16:01:40.577860 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.577843 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7q9g5" Aug 04 16:01:40.579831 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:40.579814 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba3d3f9_effd_4c57_a584_6a79d1fc4516.slice/crio-42efd4a60f32366d2f9f6378aaa42d72b3d8f8d3546b1680dca6841769d1ac7a WatchSource:0}: Error finding container 42efd4a60f32366d2f9f6378aaa42d72b3d8f8d3546b1680dca6841769d1ac7a: Status 404 returned error can't find the container with id 42efd4a60f32366d2f9f6378aaa42d72b3d8f8d3546b1680dca6841769d1ac7a Aug 04 16:01:40.584254 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:01:40.584237 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b99219_a0ce_4d7b_a976_aa710a8fa6bf.slice/crio-53e2566a1177dec473d44674b6c0d73e115b112344efa14b0339a353b2833772 WatchSource:0}: Error finding container 53e2566a1177dec473d44674b6c0d73e115b112344efa14b0339a353b2833772: Status 404 returned error can't find the container with id 53e2566a1177dec473d44674b6c0d73e115b112344efa14b0339a353b2833772 Aug 04 16:01:40.792509 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.792429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs\") pod \"network-metrics-daemon-t79jc\" (UID: \"56a1a012-b25b-4c19-b6be-73f382416c22\") " pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:40.792697 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.792583 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 16:01:40.792697 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.792655 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs podName:56a1a012-b25b-4c19-b6be-73f382416c22 nodeName:}" failed. No retries permitted until 2026-08-04 16:01:41.792635778 +0000 UTC m=+2.936968004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs") pod "network-metrics-daemon-t79jc" (UID: "56a1a012-b25b-4c19-b6be-73f382416c22") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 16:01:40.893041 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.893013 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret\") pod \"global-pull-secret-syncer-5hhk7\" (UID: \"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db\") " pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:40.893180 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.893165 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 16:01:40.893248 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.893235 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret podName:cb9271e1-f3f4-4afc-b1ce-22d79e60a3db nodeName:}" failed. No retries permitted until 2026-08-04 16:01:41.893216031 +0000 UTC m=+3.037548270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret") pod "global-pull-secret-syncer-5hhk7" (UID: "cb9271e1-f3f4-4afc-b1ce-22d79e60a3db") : object "kube-system"/"original-pull-secret" not registered Aug 04 16:01:40.993614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:40.993575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhj2\" (UniqueName: \"kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2\") pod \"network-check-target-qvmnx\" (UID: \"fc84cbe7-6fbf-4152-9bcb-6227c4126473\") " pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:40.993766 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.993726 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 16:01:40.993766 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.993748 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 16:01:40.993766 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.993761 2580 projected.go:196] Error preparing data for projected volume kube-api-access-qxhj2 for pod openshift-network-diagnostics/network-check-target-qvmnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 16:01:40.993922 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:40.993816 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2 podName:fc84cbe7-6fbf-4152-9bcb-6227c4126473 nodeName:}" failed. No retries permitted until 2026-08-04 16:01:41.993796207 +0000 UTC m=+3.138128433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qxhj2" (UniqueName: "kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2") pod "network-check-target-qvmnx" (UID: "fc84cbe7-6fbf-4152-9bcb-6227c4126473") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 16:01:41.223055 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.222905 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 15:56:40 +0000 UTC" deadline="2028-05-06 03:15:11.615187354 +0000 UTC" Aug 04 16:01:41.223055 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.222965 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15371h13m30.392226143s" Aug 04 16:01:41.313716 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.313688 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:41.313877 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:41.313803 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:01:41.320802 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.320756 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7q9g5" event={"ID":"14b99219-a0ce-4d7b-a976-aa710a8fa6bf","Type":"ContainerStarted","Data":"53e2566a1177dec473d44674b6c0d73e115b112344efa14b0339a353b2833772"} Aug 04 16:01:41.323014 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.322986 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kknn8" event={"ID":"dba3d3f9-effd-4c57-a584-6a79d1fc4516","Type":"ContainerStarted","Data":"42efd4a60f32366d2f9f6378aaa42d72b3d8f8d3546b1680dca6841769d1ac7a"} Aug 04 16:01:41.324491 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.324446 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nppcn" event={"ID":"29da60cb-4da1-44b2-a158-3532bf7c5c0d","Type":"ContainerStarted","Data":"9eb6319e34f969b7e99eda9f46dd7b3742343f4843af4ee2b4605a497dd0cc7a"} Aug 04 16:01:41.328022 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.327967 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g2845" event={"ID":"add717dd-d023-43b7-bf8b-f50516f053db","Type":"ContainerStarted","Data":"9c7684c651be4f8588fa545563aad29b40178e90b1d1925a8adf3b8673b27ba8"} Aug 04 16:01:41.330548 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.330497 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" event={"ID":"1b623ab2-5a3b-47a7-accb-ce513e36b597","Type":"ContainerStarted","Data":"3da94ad68ea6ea8e0427daa76d205182f29dcccfd4ff88ea8694668b8e79fa1f"} Aug 04 16:01:41.332853 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.332828 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" event={"ID":"0710ce46-7acc-47cf-8182-b1248d751567","Type":"ContainerStarted","Data":"6a5db26e5174a86b2036fe8a8c55604cc4155f913221a6c08b9fcd45513a9b4d"} Aug 04 16:01:41.335154 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.335096 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sdq7" event={"ID":"a9c0f306-ee82-4a4a-9680-5b0835964025","Type":"ContainerStarted","Data":"cf60898594c8e69c99411b891ab378d98b52aceb4cebcf6949c703e11761b701"} Aug 04 16:01:41.338068 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.338046 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-42.ec2.internal" event={"ID":"c8a12c8c42b46b94e7355712b1c24093","Type":"ContainerStarted","Data":"6d86e1d69e034edd93c16c401f0849c79e17c6cd3873f694fa1aa017e450ce00"} Aug 04 16:01:41.341564 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.341495 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vbhgz" event={"ID":"56b8e09d-90bb-401d-96a6-20d691168fc3","Type":"ContainerStarted","Data":"6bc37e9713318263c6f3b7dd2a3bcae932165bc80dddfe4010b6bf3902022c7e"} Aug 04 16:01:41.342990 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.342923 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lnn4g" event={"ID":"656bb7e0-4acf-401f-a87a-209fa05614fa","Type":"ContainerStarted","Data":"28884941569f4c3d94f63a88dbe1e5b2d52a062b072901d683890989cac8e527"} Aug 04 16:01:41.345813 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.345785 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal" event={"ID":"b7188073945aff2aff7853ad3566ec4f","Type":"ContainerStarted","Data":"c3334c0565733621e749eb59c352e9cc7bee3a08e3a3e9b26b5dcb706b3a5c9f"} Aug 04 16:01:41.456060 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.456035 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 16:01:41.798960 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.798924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs\") pod \"network-metrics-daemon-t79jc\" (UID: \"56a1a012-b25b-4c19-b6be-73f382416c22\") " pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:41.799134 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:41.799106 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 16:01:41.799202 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:41.799181 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs podName:56a1a012-b25b-4c19-b6be-73f382416c22 nodeName:}" failed. No retries permitted until 2026-08-04 16:01:43.799160895 +0000 UTC m=+4.943493138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs") pod "network-metrics-daemon-t79jc" (UID: "56a1a012-b25b-4c19-b6be-73f382416c22") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 16:01:41.899811 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:41.899778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret\") pod \"global-pull-secret-syncer-5hhk7\" (UID: \"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db\") " pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:41.899989 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:41.899976 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 16:01:41.900051 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:41.900037 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret podName:cb9271e1-f3f4-4afc-b1ce-22d79e60a3db nodeName:}" failed. No retries permitted until 2026-08-04 16:01:43.900019993 +0000 UTC m=+5.044352237 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret") pod "global-pull-secret-syncer-5hhk7" (UID: "cb9271e1-f3f4-4afc-b1ce-22d79e60a3db") : object "kube-system"/"original-pull-secret" not registered Aug 04 16:01:42.000544 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:42.000520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhj2\" (UniqueName: \"kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2\") pod \"network-check-target-qvmnx\" (UID: \"fc84cbe7-6fbf-4152-9bcb-6227c4126473\") " pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:42.000691 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:42.000670 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 16:01:42.000751 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:42.000695 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 16:01:42.000751 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:42.000707 2580 projected.go:196] Error preparing data for projected volume kube-api-access-qxhj2 for pod openshift-network-diagnostics/network-check-target-qvmnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 16:01:42.000848 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:42.000757 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2 podName:fc84cbe7-6fbf-4152-9bcb-6227c4126473 nodeName:}" failed. No retries permitted until 2026-08-04 16:01:44.000739287 +0000 UTC m=+5.145071538 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qxhj2" (UniqueName: "kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2") pod "network-check-target-qvmnx" (UID: "fc84cbe7-6fbf-4152-9bcb-6227c4126473") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 16:01:42.223153 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:42.223067 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 15:56:40 +0000 UTC" deadline="2028-02-09 08:15:54.12215512 +0000 UTC" Aug 04 16:01:42.223153 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:42.223102 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13288h14m11.899058763s" Aug 04 16:01:42.312996 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:42.312970 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:42.313114 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:42.313089 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvmnx" podUID="fc84cbe7-6fbf-4152-9bcb-6227c4126473" Aug 04 16:01:42.313196 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:42.313167 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:42.313309 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:42.313246 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79jc" podUID="56a1a012-b25b-4c19-b6be-73f382416c22" Aug 04 16:01:42.350206 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:42.350163 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal" event={"ID":"b7188073945aff2aff7853ad3566ec4f","Type":"ContainerStarted","Data":"0ab59e888b31d9fae1b5b647c7d3f04ccfff40d7432ad290d3a73eba71c710d5"} Aug 04 16:01:43.314091 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:43.313825 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:43.314529 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:43.314165 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:01:43.354717 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:43.354686 2580 generic.go:356] "Generic (PLEG): container finished" podID="b7188073945aff2aff7853ad3566ec4f" containerID="0ab59e888b31d9fae1b5b647c7d3f04ccfff40d7432ad290d3a73eba71c710d5" exitCode=0 Aug 04 16:01:43.354864 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:43.354727 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal" event={"ID":"b7188073945aff2aff7853ad3566ec4f","Type":"ContainerDied","Data":"0ab59e888b31d9fae1b5b647c7d3f04ccfff40d7432ad290d3a73eba71c710d5"} Aug 04 16:01:43.813595 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:43.813561 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs\") pod \"network-metrics-daemon-t79jc\" (UID: \"56a1a012-b25b-4c19-b6be-73f382416c22\") " pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:43.813741 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:43.813690 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 16:01:43.813795 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:43.813752 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs podName:56a1a012-b25b-4c19-b6be-73f382416c22 nodeName:}" failed. No retries permitted until 2026-08-04 16:01:47.813733841 +0000 UTC m=+8.958066065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs") pod "network-metrics-daemon-t79jc" (UID: "56a1a012-b25b-4c19-b6be-73f382416c22") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 16:01:43.914192 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:43.914155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret\") pod \"global-pull-secret-syncer-5hhk7\" (UID: \"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db\") " pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:43.914367 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:43.914318 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 16:01:43.914435 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:43.914381 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret podName:cb9271e1-f3f4-4afc-b1ce-22d79e60a3db nodeName:}" failed. No retries permitted until 2026-08-04 16:01:47.914365905 +0000 UTC m=+9.058698130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret") pod "global-pull-secret-syncer-5hhk7" (UID: "cb9271e1-f3f4-4afc-b1ce-22d79e60a3db") : object "kube-system"/"original-pull-secret" not registered Aug 04 16:01:44.015213 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:44.015153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhj2\" (UniqueName: \"kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2\") pod \"network-check-target-qvmnx\" (UID: \"fc84cbe7-6fbf-4152-9bcb-6227c4126473\") " pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:44.015379 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:44.015318 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 16:01:44.015379 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:44.015337 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 16:01:44.015379 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:44.015349 2580 projected.go:196] Error preparing data for projected volume kube-api-access-qxhj2 for pod openshift-network-diagnostics/network-check-target-qvmnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 16:01:44.015515 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:44.015404 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2 podName:fc84cbe7-6fbf-4152-9bcb-6227c4126473 nodeName:}" failed. No retries permitted until 2026-08-04 16:01:48.015389369 +0000 UTC m=+9.159721594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qxhj2" (UniqueName: "kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2") pod "network-check-target-qvmnx" (UID: "fc84cbe7-6fbf-4152-9bcb-6227c4126473") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 16:01:44.313603 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:44.313574 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:44.313753 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:44.313702 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvmnx" podUID="fc84cbe7-6fbf-4152-9bcb-6227c4126473" Aug 04 16:01:44.313813 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:44.313750 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:44.313871 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:44.313849 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79jc" podUID="56a1a012-b25b-4c19-b6be-73f382416c22" Aug 04 16:01:45.315822 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:45.315341 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:45.315822 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:45.315468 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:01:45.363037 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:45.362997 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal" event={"ID":"b7188073945aff2aff7853ad3566ec4f","Type":"ContainerStarted","Data":"cc51eb3a60363725a4ef4990a3d0d5c5d05b6dfc761c9c87cee20932a41b6f0f"} Aug 04 16:01:45.372875 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:45.372845 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g2845" event={"ID":"add717dd-d023-43b7-bf8b-f50516f053db","Type":"ContainerStarted","Data":"4da835867ceed51f26f0cb1a872ce84a9c2d260abba4d023e23564494320dc40"} Aug 04 16:01:45.389177 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:45.389149 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" event={"ID":"1b623ab2-5a3b-47a7-accb-ce513e36b597","Type":"ContainerStarted","Data":"048d9e968845554ed4d80bcce0369f4abf866b1006e053c03b1432282a44f48f"} Aug 04 16:01:45.399621 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:45.399574 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g2845" podStartSLOduration=2.267870087 podStartE2EDuration="6.399557148s" totalImagesPullingTime="4.131687061s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 16:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:01:45.399274816 +0000 UTC m=+6.543607062" watchObservedRunningTime="2026-08-04 16:01:45.399557148 +0000 UTC m=+6.543889395" Aug 04 16:01:45.400361 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:45.400329 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-42.ec2.internal" podStartSLOduration=5.400320418 podStartE2EDuration="5.400320418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 16:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:01:45.382099559 +0000 UTC m=+6.526431828" watchObservedRunningTime="2026-08-04 16:01:45.400320418 +0000 UTC m=+6.544652664" Aug 04 16:01:46.313184 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:46.313153 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:46.313367 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:46.313348 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:46.313506 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:46.313441 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79jc" podUID="56a1a012-b25b-4c19-b6be-73f382416c22" Aug 04 16:01:46.313506 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:46.313274 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvmnx" podUID="fc84cbe7-6fbf-4152-9bcb-6227c4126473" Aug 04 16:01:47.313065 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:47.312975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:47.313501 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:47.313095 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:01:47.846431 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:47.846393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs\") pod \"network-metrics-daemon-t79jc\" (UID: \"56a1a012-b25b-4c19-b6be-73f382416c22\") " pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:47.846660 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:47.846591 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 16:01:47.846660 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:47.846659 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs podName:56a1a012-b25b-4c19-b6be-73f382416c22 nodeName:}" failed. No retries permitted until 2026-08-04 16:01:55.846640051 +0000 UTC m=+16.990972289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs") pod "network-metrics-daemon-t79jc" (UID: "56a1a012-b25b-4c19-b6be-73f382416c22") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 16:01:47.947006 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:47.946879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret\") pod \"global-pull-secret-syncer-5hhk7\" (UID: \"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db\") " pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:47.947178 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:47.947031 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 16:01:47.947178 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:47.947088 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret podName:cb9271e1-f3f4-4afc-b1ce-22d79e60a3db nodeName:}" failed. No retries permitted until 2026-08-04 16:01:55.947071132 +0000 UTC m=+17.091403360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret") pod "global-pull-secret-syncer-5hhk7" (UID: "cb9271e1-f3f4-4afc-b1ce-22d79e60a3db") : object "kube-system"/"original-pull-secret" not registered Aug 04 16:01:48.047613 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:48.047478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhj2\" (UniqueName: \"kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2\") pod \"network-check-target-qvmnx\" (UID: \"fc84cbe7-6fbf-4152-9bcb-6227c4126473\") " pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:48.047779 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:48.047659 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 16:01:48.047779 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:48.047685 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 16:01:48.047779 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:48.047698 2580 projected.go:196] Error preparing data for projected volume kube-api-access-qxhj2 for pod openshift-network-diagnostics/network-check-target-qvmnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 16:01:48.047779 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:48.047755 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2 podName:fc84cbe7-6fbf-4152-9bcb-6227c4126473 nodeName:}" failed. No retries permitted until 2026-08-04 16:01:56.04773745 +0000 UTC m=+17.192069679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qxhj2" (UniqueName: "kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2") pod "network-check-target-qvmnx" (UID: "fc84cbe7-6fbf-4152-9bcb-6227c4126473") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 16:01:48.313574 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:48.313496 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:48.313989 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:48.313633 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvmnx" podUID="fc84cbe7-6fbf-4152-9bcb-6227c4126473" Aug 04 16:01:48.313989 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:48.313714 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:48.313989 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:48.313830 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79jc" podUID="56a1a012-b25b-4c19-b6be-73f382416c22" Aug 04 16:01:49.315407 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:49.314893 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:49.315407 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:49.315004 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:01:50.313472 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:50.313441 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:50.313706 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:50.313581 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79jc" podUID="56a1a012-b25b-4c19-b6be-73f382416c22" Aug 04 16:01:50.313706 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:50.313445 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:50.313837 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:50.313726 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvmnx" podUID="fc84cbe7-6fbf-4152-9bcb-6227c4126473" Aug 04 16:01:51.313165 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:51.313063 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:51.313739 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:51.313219 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:01:52.313852 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:52.313823 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:52.313852 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:52.313834 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:52.314353 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:52.313928 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvmnx" podUID="fc84cbe7-6fbf-4152-9bcb-6227c4126473" Aug 04 16:01:52.314353 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:52.314003 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79jc" podUID="56a1a012-b25b-4c19-b6be-73f382416c22" Aug 04 16:01:53.312958 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:53.312923 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:53.313119 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:53.313052 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:01:54.313746 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:54.313711 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:54.313746 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:54.313753 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:54.314246 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:54.313842 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvmnx" podUID="fc84cbe7-6fbf-4152-9bcb-6227c4126473" Aug 04 16:01:54.314246 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:54.313953 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79jc" podUID="56a1a012-b25b-4c19-b6be-73f382416c22" Aug 04 16:01:55.313909 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:55.313835 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:55.314334 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:55.313972 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:01:55.908170 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:55.908118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs\") pod \"network-metrics-daemon-t79jc\" (UID: \"56a1a012-b25b-4c19-b6be-73f382416c22\") " pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:55.908387 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:55.908311 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 16:01:55.908439 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:55.908390 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs podName:56a1a012-b25b-4c19-b6be-73f382416c22 nodeName:}" failed. No retries permitted until 2026-08-04 16:02:11.908368263 +0000 UTC m=+33.052700504 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs") pod "network-metrics-daemon-t79jc" (UID: "56a1a012-b25b-4c19-b6be-73f382416c22") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 16:01:56.009479 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:56.009450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret\") pod \"global-pull-secret-syncer-5hhk7\" (UID: \"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db\") " pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:56.009645 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:56.009567 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 16:01:56.009645 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:56.009621 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret podName:cb9271e1-f3f4-4afc-b1ce-22d79e60a3db nodeName:}" failed. No retries permitted until 2026-08-04 16:02:12.009606965 +0000 UTC m=+33.153939189 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret") pod "global-pull-secret-syncer-5hhk7" (UID: "cb9271e1-f3f4-4afc-b1ce-22d79e60a3db") : object "kube-system"/"original-pull-secret" not registered Aug 04 16:01:56.110009 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:56.109978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhj2\" (UniqueName: \"kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2\") pod \"network-check-target-qvmnx\" (UID: \"fc84cbe7-6fbf-4152-9bcb-6227c4126473\") " pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:56.110167 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:56.110118 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 16:01:56.110167 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:56.110141 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 16:01:56.110167 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:56.110151 2580 projected.go:196] Error preparing data for projected volume kube-api-access-qxhj2 for pod openshift-network-diagnostics/network-check-target-qvmnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 16:01:56.110308 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:56.110195 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2 podName:fc84cbe7-6fbf-4152-9bcb-6227c4126473 nodeName:}" failed. No retries permitted until 2026-08-04 16:02:12.110182499 +0000 UTC m=+33.254514728 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qxhj2" (UniqueName: "kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2") pod "network-check-target-qvmnx" (UID: "fc84cbe7-6fbf-4152-9bcb-6227c4126473") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 16:01:56.313892 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:56.313824 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:56.314101 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:56.313949 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvmnx" podUID="fc84cbe7-6fbf-4152-9bcb-6227c4126473" Aug 04 16:01:56.314101 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:56.313981 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:56.314101 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:56.314090 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79jc" podUID="56a1a012-b25b-4c19-b6be-73f382416c22" Aug 04 16:01:57.312964 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:57.312932 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:57.313133 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:57.313049 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:01:58.313441 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:58.313412 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:01:58.313847 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:58.313412 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:01:58.313847 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:58.313548 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79jc" podUID="56a1a012-b25b-4c19-b6be-73f382416c22" Aug 04 16:01:58.313847 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:58.313595 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvmnx" podUID="fc84cbe7-6fbf-4152-9bcb-6227c4126473" Aug 04 16:01:59.313867 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:59.313779 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:01:59.314173 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:01:59.313887 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:01:59.555609 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:01:59.555403 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 16:02:00.240975 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.240737 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T16:01:59.55560637Z","UUID":"1bd93111-92e4-446f-ba13-71a4b16d91b9","Handler":null,"Name":"","Endpoint":""} Aug 04 16:02:00.242005 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.241987 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 16:02:00.242082 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.242020 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 16:02:00.313334 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.313264 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:02:00.313334 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.313316 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:02:00.313494 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:00.313400 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvmnx" podUID="fc84cbe7-6fbf-4152-9bcb-6227c4126473" Aug 04 16:02:00.313543 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:00.313523 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79jc" podUID="56a1a012-b25b-4c19-b6be-73f382416c22" Aug 04 16:02:00.419721 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.419685 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7q9g5" event={"ID":"14b99219-a0ce-4d7b-a976-aa710a8fa6bf","Type":"ContainerStarted","Data":"eebb0a9c4940bc6292691b645a75aa79102f38240ee50a689364fb1d2ffba35e"} Aug 04 16:02:00.421009 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.420989 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kknn8" event={"ID":"dba3d3f9-effd-4c57-a584-6a79d1fc4516","Type":"ContainerStarted","Data":"05aa9ce15e0ebff0dddf7052b7714919bbf1b6cac838a5d3999c1ba79f599c97"} Aug 04 16:02:00.422656 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.422626 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" event={"ID":"1b623ab2-5a3b-47a7-accb-ce513e36b597","Type":"ContainerStarted","Data":"3da6212cdf911b228abdf32fee7fea73649facefed5aa71c9ccafdd9e49c17fe"} Aug 04 16:02:00.425362 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.425341 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" event={"ID":"0710ce46-7acc-47cf-8182-b1248d751567","Type":"ContainerStarted","Data":"3728c61b3699cd2f8b1bca1ff21cedef441e2d9d17c8ec313aa0f1dd22b84658"} Aug 04 16:02:00.425465 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.425398 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" event={"ID":"0710ce46-7acc-47cf-8182-b1248d751567","Type":"ContainerStarted","Data":"c4f68eee6c8c36bc6df90492773c890401b6ab5c6c77488116a63db32f022be3"} Aug 04 16:02:00.425465 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.425434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" event={"ID":"0710ce46-7acc-47cf-8182-b1248d751567","Type":"ContainerStarted","Data":"1e16a2639e9a9f84fcc889dd3bf072e8e945e4f47058963006a9ee59587bc4a6"} Aug 04 16:02:00.425610 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.425475 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" event={"ID":"0710ce46-7acc-47cf-8182-b1248d751567","Type":"ContainerStarted","Data":"284c88e7cdf14037eaa392a7264f36800e006a344cd9bbea8025756df0039226"} Aug 04 16:02:00.425610 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.425505 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" event={"ID":"0710ce46-7acc-47cf-8182-b1248d751567","Type":"ContainerStarted","Data":"3fc19a7dc7c54a93272718c0390f8d71be53a46136624ae95de89efb40fb058f"} Aug 04 16:02:00.425610 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.425549 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" event={"ID":"0710ce46-7acc-47cf-8182-b1248d751567","Type":"ContainerStarted","Data":"091ec20b148bfd87e56ff11f0335b16dae623d817d7e62d54723d14103c0a55c"} Aug 04 16:02:00.426658 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.426637 2580 generic.go:356] "Generic (PLEG): container finished" podID="a9c0f306-ee82-4a4a-9680-5b0835964025" containerID="e39fae59b9cdd2dcb2e17da3c60863d818cbda29cad3b848710a3a4c4a8ccab6" exitCode=0 Aug 04 16:02:00.426739 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.426707 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sdq7" event={"ID":"a9c0f306-ee82-4a4a-9680-5b0835964025","Type":"ContainerDied","Data":"e39fae59b9cdd2dcb2e17da3c60863d818cbda29cad3b848710a3a4c4a8ccab6"} Aug 04 16:02:00.428031 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.428020 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-42.ec2.internal" event={"ID":"c8a12c8c42b46b94e7355712b1c24093","Type":"ContainerStarted","Data":"d303c67ca6feb061516329dd68393980af535d85db52553173fc74dac85d31b9"} Aug 04 16:02:00.429357 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.429322 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vbhgz" event={"ID":"56b8e09d-90bb-401d-96a6-20d691168fc3","Type":"ContainerStarted","Data":"b14fa176811a9beedf6ad1ebcaee73d52c6182596d39d08867f88a54845a80e8"} Aug 04 16:02:00.430632 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.430613 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lnn4g" event={"ID":"656bb7e0-4acf-401f-a87a-209fa05614fa","Type":"ContainerStarted","Data":"2543650ba3ed238d2d95682e0827dd29ae90517dece377e4f0fd65569f5d72f0"} Aug 04 16:02:00.439653 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.439620 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7q9g5" podStartSLOduration=2.40220994 podStartE2EDuration="21.439608454s" totalImagesPullingTime="19.037398514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 16:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:00.438995523 +0000 UTC m=+21.583327782" watchObservedRunningTime="2026-08-04 16:02:00.439608454 +0000 UTC m=+21.583940700" Aug 04 16:02:00.455252 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.455222 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vbhgz" podStartSLOduration=2.6924927309999998 podStartE2EDuration="21.455211826s" totalImagesPullingTime="18.762719095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 16:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:00.455018664 +0000 UTC m=+21.599350910" watchObservedRunningTime="2026-08-04 16:02:00.455211826 +0000 UTC m=+21.599544071" Aug 04 16:02:00.495225 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.495191 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lnn4g" podStartSLOduration=2.691544921 podStartE2EDuration="21.495179719s" totalImagesPullingTime="18.803634798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:00.494778377 +0000 UTC m=+21.639110644" watchObservedRunningTime="2026-08-04 16:02:00.495179719 +0000 UTC m=+21.639511972" Aug 04 16:02:00.512084 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.512020 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kknn8" podStartSLOduration=2.76680172 podStartE2EDuration="21.512010212s" totalImagesPullingTime="18.745208492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 16:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:00.511855891 +0000 UTC m=+21.656188139" watchObservedRunningTime="2026-08-04 16:02:00.512010212 +0000 UTC m=+21.656342456" Aug 04 16:02:00.527338 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:00.527304 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-42.ec2.internal" podStartSLOduration=20.527276735 podStartE2EDuration="20.527276735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 16:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:00.527027028 +0000 UTC m=+21.671359285" watchObservedRunningTime="2026-08-04 16:02:00.527276735 +0000 UTC m=+21.671608980" Aug 04 16:02:01.313867 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:01.313842 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:02:01.314063 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:01.313933 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:02:01.434433 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:01.434408 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" event={"ID":"1b623ab2-5a3b-47a7-accb-ce513e36b597","Type":"ContainerStarted","Data":"e17b9deae4713bda31fde76bdb7254dc4ed7726582ffd4d66696cede5c0498b5"} Aug 04 16:02:01.435929 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:01.435903 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nppcn" event={"ID":"29da60cb-4da1-44b2-a158-3532bf7c5c0d","Type":"ContainerStarted","Data":"2bc8a3dc56ee4467dc599fbc8fb238c468e563ed380812b4671ddf9590a7d79f"} Aug 04 16:02:01.461772 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:01.460650 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb97" podStartSLOduration=2.584393685 podStartE2EDuration="22.460630907s" totalImagesPullingTime="19.876237222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 16:01:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:01.458094857 +0000 UTC m=+22.602427103" watchObservedRunningTime="2026-08-04 16:02:01.460630907 +0000 UTC m=+22.604963152" Aug 04 16:02:02.313170 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:02.313143 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:02:02.313311 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:02.313152 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:02:02.313311 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:02.313227 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvmnx" podUID="fc84cbe7-6fbf-4152-9bcb-6227c4126473" Aug 04 16:02:02.313399 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:02.313356 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79jc" podUID="56a1a012-b25b-4c19-b6be-73f382416c22" Aug 04 16:02:02.440493 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:02.440434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" event={"ID":"0710ce46-7acc-47cf-8182-b1248d751567","Type":"ContainerStarted","Data":"bf1ae31e04115668fa46bd24686e6fbd6a854eccba06d96f444a1be07107acd5"} Aug 04 16:02:03.313328 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:03.313299 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:02:03.313520 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:03.313395 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:02:04.313785 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:04.313643 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:02:04.314214 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:04.313717 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:02:04.314214 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:04.313863 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79jc" podUID="56a1a012-b25b-4c19-b6be-73f382416c22" Aug 04 16:02:04.314214 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:04.313979 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvmnx" podUID="fc84cbe7-6fbf-4152-9bcb-6227c4126473" Aug 04 16:02:04.447687 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:04.447655 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" event={"ID":"0710ce46-7acc-47cf-8182-b1248d751567","Type":"ContainerStarted","Data":"995ea96121ab0e5b08da0a783f05fbadd92402d618f37ed6c5aa0ad0b4847d5e"} Aug 04 16:02:04.447902 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:04.447880 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:02:04.462263 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:04.462242 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:02:04.484997 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:04.484929 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nppcn" podStartSLOduration=6.716063614 podStartE2EDuration="25.484914794s" totalImagesPullingTime="18.76885118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 16:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:01.478440571 +0000 UTC m=+22.622772817" watchObservedRunningTime="2026-08-04 16:02:04.484914794 +0000 UTC m=+25.629247038" Aug 04 16:02:04.537423 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:04.537367 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" podStartSLOduration=6.712142571 podStartE2EDuration="25.537355842s" totalImagesPullingTime="18.825213271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 16:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:04.49106357 +0000 UTC m=+25.635395838" watchObservedRunningTime="2026-08-04 16:02:04.537355842 +0000 UTC m=+25.681688090" Aug 04 16:02:04.560198 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:04.560181 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lnn4g" Aug 04 16:02:04.560709 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:04.560694 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lnn4g" Aug 04 16:02:05.313032 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:05.313005 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:02:05.313200 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:05.313132 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:02:05.455641 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:05.455406 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lnn4g" Aug 04 16:02:05.456204 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:05.455649 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:02:05.456204 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:05.455701 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:02:05.456204 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:05.455767 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lnn4g" Aug 04 16:02:05.473930 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:05.473904 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:02:06.229318 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:06.229095 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5hhk7"] Aug 04 16:02:06.229499 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:06.229402 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:02:06.229570 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:06.229501 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:02:06.233323 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:06.233162 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t79jc"] Aug 04 16:02:06.233323 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:06.233264 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:02:06.233631 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:06.233589 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79jc" podUID="56a1a012-b25b-4c19-b6be-73f382416c22" Aug 04 16:02:06.234275 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:06.234255 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qvmnx"] Aug 04 16:02:06.234436 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:06.234417 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:02:06.234564 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:06.234521 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvmnx" podUID="fc84cbe7-6fbf-4152-9bcb-6227c4126473" Aug 04 16:02:08.312978 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:08.312918 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:02:08.312978 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:08.312934 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:02:08.312978 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:08.312943 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:02:08.313537 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:08.313060 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79jc" podUID="56a1a012-b25b-4c19-b6be-73f382416c22" Aug 04 16:02:08.313537 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:08.313192 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvmnx" podUID="fc84cbe7-6fbf-4152-9bcb-6227c4126473" Aug 04 16:02:08.313537 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:08.313293 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:02:09.464798 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:09.464771 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sdq7" event={"ID":"a9c0f306-ee82-4a4a-9680-5b0835964025","Type":"ContainerStarted","Data":"5eacd461811b2a20a3cb28f5ebef91b7645523fb34193eb7e924fea806693255"} Aug 04 16:02:10.313541 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:10.313463 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:02:10.313679 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:10.313551 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:02:10.313679 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:10.313563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:02:10.313679 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:10.313558 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvmnx" podUID="fc84cbe7-6fbf-4152-9bcb-6227c4126473" Aug 04 16:02:10.313679 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:10.313622 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5hhk7" podUID="cb9271e1-f3f4-4afc-b1ce-22d79e60a3db" Aug 04 16:02:10.313825 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:10.313716 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79jc" podUID="56a1a012-b25b-4c19-b6be-73f382416c22" Aug 04 16:02:10.468266 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:10.468098 2580 generic.go:356] "Generic (PLEG): container finished" podID="a9c0f306-ee82-4a4a-9680-5b0835964025" containerID="5eacd461811b2a20a3cb28f5ebef91b7645523fb34193eb7e924fea806693255" exitCode=0 Aug 04 16:02:10.468266 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:10.468177 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sdq7" event={"ID":"a9c0f306-ee82-4a4a-9680-5b0835964025","Type":"ContainerDied","Data":"5eacd461811b2a20a3cb28f5ebef91b7645523fb34193eb7e924fea806693255"} Aug 04 16:02:11.935456 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:11.935428 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs\") pod \"network-metrics-daemon-t79jc\" (UID: \"56a1a012-b25b-4c19-b6be-73f382416c22\") " pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:02:11.935877 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:11.935560 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 16:02:11.935877 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:11.935613 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs podName:56a1a012-b25b-4c19-b6be-73f382416c22 nodeName:}" failed. No retries permitted until 2026-08-04 16:02:43.935600409 +0000 UTC m=+65.079932636 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs") pod "network-metrics-daemon-t79jc" (UID: "56a1a012-b25b-4c19-b6be-73f382416c22") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 16:02:12.036194 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.036164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret\") pod \"global-pull-secret-syncer-5hhk7\" (UID: \"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db\") " pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:02:12.036332 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.036317 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 16:02:12.036379 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.036372 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret podName:cb9271e1-f3f4-4afc-b1ce-22d79e60a3db nodeName:}" failed. No retries permitted until 2026-08-04 16:02:44.036356297 +0000 UTC m=+65.180688532 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret") pod "global-pull-secret-syncer-5hhk7" (UID: "cb9271e1-f3f4-4afc-b1ce-22d79e60a3db") : object "kube-system"/"original-pull-secret" not registered Aug 04 16:02:12.137473 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.137447 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhj2\" (UniqueName: \"kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2\") pod \"network-check-target-qvmnx\" (UID: \"fc84cbe7-6fbf-4152-9bcb-6227c4126473\") " pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:02:12.137613 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.137551 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 16:02:12.137613 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.137565 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 16:02:12.137613 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.137573 2580 projected.go:196] Error preparing data for projected volume kube-api-access-qxhj2 for pod openshift-network-diagnostics/network-check-target-qvmnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 16:02:12.137717 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.137616 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2 podName:fc84cbe7-6fbf-4152-9bcb-6227c4126473 nodeName:}" failed. No retries permitted until 2026-08-04 16:02:44.137603487 +0000 UTC m=+65.281935711 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qxhj2" (UniqueName: "kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2") pod "network-check-target-qvmnx" (UID: "fc84cbe7-6fbf-4152-9bcb-6227c4126473") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 16:02:12.190218 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.190170 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-42.ec2.internal" event="NodeReady" Aug 04 16:02:12.190341 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.190263 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 16:02:12.227490 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.227469 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn"] Aug 04 16:02:12.244210 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.244188 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p"] Aug 04 16:02:12.244341 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.244305 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn" Aug 04 16:02:12.248811 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.248786 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Aug 04 16:02:12.249683 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.248940 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-grjqs\"" Aug 04 16:02:12.249683 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.249184 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Aug 04 16:02:12.249683 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.249195 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Aug 04 16:02:12.249683 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.249454 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Aug 04 16:02:12.258572 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.258554 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr"] Aug 04 16:02:12.258679 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.258667 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.261629 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.261608 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s5xs2\"" Aug 04 16:02:12.261721 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.261608 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 04 16:02:12.261721 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.261670 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 04 16:02:12.262145 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.262131 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 04 16:02:12.269049 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.269029 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 04 16:02:12.272888 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.272870 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv"] Aug 04 16:02:12.272984 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.272970 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:12.275519 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.275469 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Aug 04 16:02:12.275519 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.275473 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 16:02:12.275694 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.275544 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 16:02:12.275818 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.275802 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cr8tk\"" Aug 04 16:02:12.276183 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.276162 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Aug 04 16:02:12.287412 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.287393 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-jcbjw"] Aug 04 16:02:12.287511 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.287498 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" Aug 04 16:02:12.289995 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.289965 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Aug 04 16:02:12.290096 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.290007 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Aug 04 16:02:12.290096 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.290055 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Aug 04 16:02:12.290275 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.290257 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6nvhd\"" Aug 04 16:02:12.303744 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.303722 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp"] Aug 04 16:02:12.303963 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.303950 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:12.306962 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.306939 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Aug 04 16:02:12.307056 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.307016 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-j5pfr\"" Aug 04 16:02:12.307265 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.307252 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Aug 04 16:02:12.307452 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.307440 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Aug 04 16:02:12.307884 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.307869 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Aug 04 16:02:12.312838 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.312823 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Aug 04 16:02:12.319192 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.319174 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-7fvfv"] Aug 04 16:02:12.319304 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.319275 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp" Aug 04 16:02:12.321673 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.321658 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Aug 04 16:02:12.321825 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.321811 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Aug 04 16:02:12.322011 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.321999 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Aug 04 16:02:12.322046 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.322008 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Aug 04 16:02:12.322081 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.322042 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2gddf\"" Aug 04 16:02:12.328349 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.328332 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn"] Aug 04 16:02:12.328438 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.328363 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp"] Aug 04 16:02:12.328438 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.328396 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-7fvfv" Aug 04 16:02:12.328438 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.328411 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:02:12.328438 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.328417 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:02:12.328438 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.328427 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:02:12.328665 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.328459 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-8scs4"] Aug 04 16:02:12.330954 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.330936 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Aug 04 16:02:12.331223 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.331211 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Aug 04 16:02:12.331316 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.331225 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-brgkl\"" Aug 04 16:02:12.331377 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.331303 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 16:02:12.331450 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.331430 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fv4gp\"" Aug 04 16:02:12.331605 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.331452 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kb5nq\"" Aug 04 16:02:12.331605 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.331481 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 16:02:12.331712 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.331658 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 16:02:12.331712 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.331662 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 16:02:12.338497 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.338479 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64e539db-01a3-48e9-bf35-35807cebc38f-trusted-ca\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.338586 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.338509 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64e539db-01a3-48e9-bf35-35807cebc38f-installation-pull-secrets\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.338586 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.338536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn856\" (UniqueName: \"kubernetes.io/projected/cb89b27f-9d1b-41b0-830e-1e464d29d6f1-kube-api-access-mn856\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v8nwn\" (UID: \"cb89b27f-9d1b-41b0-830e-1e464d29d6f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn" Aug 04 16:02:12.338668 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.338602 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.338668 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.338636 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64e539db-01a3-48e9-bf35-35807cebc38f-registry-certificates\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.338761 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.338667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-bound-sa-token\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.338761 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.338687 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb89b27f-9d1b-41b0-830e-1e464d29d6f1-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v8nwn\" (UID: \"cb89b27f-9d1b-41b0-830e-1e464d29d6f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn" Aug 04 16:02:12.338761 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.338734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64e539db-01a3-48e9-bf35-35807cebc38f-image-registry-private-configuration\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.338904 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.338764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvrks\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-kube-api-access-hvrks\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.338904 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.338820 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64e539db-01a3-48e9-bf35-35807cebc38f-ca-trust-extracted\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.338904 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.338855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb89b27f-9d1b-41b0-830e-1e464d29d6f1-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v8nwn\" (UID: \"cb89b27f-9d1b-41b0-830e-1e464d29d6f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn" Aug 04 16:02:12.342481 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.342453 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr"] Aug 04 16:02:12.342558 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.342492 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-7fvfv"] Aug 04 16:02:12.342558 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.342511 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-jcbjw"] Aug 04 16:02:12.342558 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.342529 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p"] Aug 04 16:02:12.342558 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.342548 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zj6tj"] Aug 04 16:02:12.342558 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.342552 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.345530 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.345510 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 16:02:12.345613 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.345546 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 16:02:12.345776 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.345761 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Aug 04 16:02:12.346353 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.346336 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vzswz\"" Aug 04 16:02:12.346505 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.346492 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Aug 04 16:02:12.352150 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.352135 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Aug 04 16:02:12.354916 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.354898 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs"] Aug 04 16:02:12.355005 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.354995 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:12.357367 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.357351 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d2pjg\"" Aug 04 16:02:12.357586 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.357574 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 16:02:12.358012 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.357998 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 16:02:12.367398 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.367376 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6d8f46644-nmgqm"] Aug 04 16:02:12.367490 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.367469 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" Aug 04 16:02:12.369836 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.369814 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xpvpq\"" Aug 04 16:02:12.369910 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.369897 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Aug 04 16:02:12.370141 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.370127 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Aug 04 16:02:12.383638 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.383620 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v8jrh"] Aug 04 16:02:12.383822 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.383803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:12.386604 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.386572 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Aug 04 16:02:12.386810 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.386780 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6jz69\"" Aug 04 16:02:12.387198 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.387174 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Aug 04 16:02:12.387887 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.387866 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 16:02:12.387975 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.387905 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Aug 04 16:02:12.388057 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.388039 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 16:02:12.388128 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.388111 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Aug 04 16:02:12.396613 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.396594 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-krplv"] Aug 04 16:02:12.396721 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.396707 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v8jrh" Aug 04 16:02:12.399292 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.399262 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 16:02:12.399388 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.399312 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 16:02:12.399388 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.399319 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-svsf8\"" Aug 04 16:02:12.399388 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.399352 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 16:02:12.412329 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.412312 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-8scs4"] Aug 04 16:02:12.412416 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.412338 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv"] Aug 04 16:02:12.412416 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.412355 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zj6tj"] Aug 04 16:02:12.412416 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.412375 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs"] Aug 04 16:02:12.412416 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.412388 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v8jrh"] Aug 04 16:02:12.412416 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.412398 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-krplv"] Aug 04 16:02:12.412416 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.412408 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d8f46644-nmgqm"] Aug 04 16:02:12.412616 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.412474 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-krplv" Aug 04 16:02:12.414949 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.414934 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5p7wf\"" Aug 04 16:02:12.439460 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.439437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64e539db-01a3-48e9-bf35-35807cebc38f-registry-certificates\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.439546 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.439465 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln589\" (UniqueName: \"kubernetes.io/projected/687ce5a4-b2e9-4c41-bb89-89d423b1e514-kube-api-access-ln589\") pod \"service-ca-operator-784f97bcf6-sg8sp\" (UID: \"687ce5a4-b2e9-4c41-bb89-89d423b1e514\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp" Aug 04 16:02:12.439546 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.439499 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-n4zpv\" (UID: \"e9647849-9d1f-4b7e-8d2b-4697dadc4dbc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" Aug 04 16:02:12.439546 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.439523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01d084ac-ce47-4d96-bbf0-2e78b83beb17-serving-cert\") pod \"console-operator-5ccd6b445f-jcbjw\" (UID: \"01d084ac-ce47-4d96-bbf0-2e78b83beb17\") " pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:12.439658 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.439555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64e539db-01a3-48e9-bf35-35807cebc38f-image-registry-private-configuration\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.439658 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.439584 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e015152-d986-4b2b-9f93-296d9291b74e-tmp\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.439658 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.439638 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01d084ac-ce47-4d96-bbf0-2e78b83beb17-config\") pod \"console-operator-5ccd6b445f-jcbjw\" (UID: \"01d084ac-ce47-4d96-bbf0-2e78b83beb17\") " pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:12.439756 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.439665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64e539db-01a3-48e9-bf35-35807cebc38f-ca-trust-extracted\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.439806 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.439777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/687ce5a4-b2e9-4c41-bb89-89d423b1e514-serving-cert\") pod \"service-ca-operator-784f97bcf6-sg8sp\" (UID: \"687ce5a4-b2e9-4c41-bb89-89d423b1e514\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp" Aug 04 16:02:12.439859 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.439830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb89b27f-9d1b-41b0-830e-1e464d29d6f1-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v8nwn\" (UID: \"cb89b27f-9d1b-41b0-830e-1e464d29d6f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn" Aug 04 16:02:12.439910 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.439860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e015152-d986-4b2b-9f93-296d9291b74e-trusted-ca-bundle\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.439910 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.439887 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c72wt\" (UniqueName: \"kubernetes.io/projected/84092957-5deb-4cae-b1ba-310f24137f8e-kube-api-access-c72wt\") pod \"cluster-monitoring-operator-6f55c4f58d-rz5dr\" (UID: \"84092957-5deb-4cae-b1ba-310f24137f8e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:12.440010 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.439913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/687ce5a4-b2e9-4c41-bb89-89d423b1e514-config\") pod \"service-ca-operator-784f97bcf6-sg8sp\" (UID: \"687ce5a4-b2e9-4c41-bb89-89d423b1e514\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp" Aug 04 16:02:12.440010 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.439938 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01d084ac-ce47-4d96-bbf0-2e78b83beb17-trusted-ca\") pod \"console-operator-5ccd6b445f-jcbjw\" (UID: \"01d084ac-ce47-4d96-bbf0-2e78b83beb17\") " pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:12.440010 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.439967 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64e539db-01a3-48e9-bf35-35807cebc38f-ca-trust-extracted\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.440010 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.439980 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e015152-d986-4b2b-9f93-296d9291b74e-serving-cert\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.440146 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440038 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/84092957-5deb-4cae-b1ba-310f24137f8e-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-rz5dr\" (UID: \"84092957-5deb-4cae-b1ba-310f24137f8e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:12.440146 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440042 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64e539db-01a3-48e9-bf35-35807cebc38f-registry-certificates\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.440146 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7sf7\" (UniqueName: \"kubernetes.io/projected/cdaafd1b-8147-4e4d-82c5-578621b9a646-kube-api-access-q7sf7\") pod \"volume-data-source-validator-5b8776669-7fvfv\" (UID: \"cdaafd1b-8147-4e4d-82c5-578621b9a646\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-7fvfv" Aug 04 16:02:12.440146 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn856\" (UniqueName: \"kubernetes.io/projected/cb89b27f-9d1b-41b0-830e-1e464d29d6f1-kube-api-access-mn856\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v8nwn\" (UID: \"cb89b27f-9d1b-41b0-830e-1e464d29d6f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn" Aug 04 16:02:12.440146 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.440456 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-bound-sa-token\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.440456 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.440210 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 16:02:12.440456 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.440223 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p: secret "image-registry-tls" not found Aug 04 16:02:12.440456 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb89b27f-9d1b-41b0-830e-1e464d29d6f1-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v8nwn\" (UID: \"cb89b27f-9d1b-41b0-830e-1e464d29d6f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn" Aug 04 16:02:12.440456 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvrks\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-kube-api-access-hvrks\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.440456 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.440275 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls podName:64e539db-01a3-48e9-bf35-35807cebc38f nodeName:}" failed. No retries permitted until 2026-08-04 16:02:12.940258715 +0000 UTC m=+34.084590945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls") pod "image-registry-7c7ddb6d8b-g9s7p" (UID: "64e539db-01a3-48e9-bf35-35807cebc38f") : secret "image-registry-tls" not found Aug 04 16:02:12.440456 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwd2p\" (UniqueName: \"kubernetes.io/projected/b557e1a0-4af6-4c29-bb50-214fa9d6882e-kube-api-access-cwd2p\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:12.440456 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e015152-d986-4b2b-9f93-296d9291b74e-service-ca-bundle\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.440456 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwv4\" (UniqueName: \"kubernetes.io/projected/8e015152-d986-4b2b-9f93-296d9291b74e-kube-api-access-qgwv4\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.440456 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8e015152-d986-4b2b-9f93-296d9291b74e-snapshots\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.440456 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440449 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:12.440456 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440469 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b557e1a0-4af6-4c29-bb50-214fa9d6882e-tmp-dir\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:12.441030 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440520 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-rz5dr\" (UID: \"84092957-5deb-4cae-b1ba-310f24137f8e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:12.441030 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmfv9\" (UniqueName: \"kubernetes.io/projected/01d084ac-ce47-4d96-bbf0-2e78b83beb17-kube-api-access-nmfv9\") pod \"console-operator-5ccd6b445f-jcbjw\" (UID: \"01d084ac-ce47-4d96-bbf0-2e78b83beb17\") " pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:12.441030 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb6p7\" (UniqueName: \"kubernetes.io/projected/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-kube-api-access-wb6p7\") pod \"cluster-samples-operator-559cbf5745-n4zpv\" (UID: \"e9647849-9d1f-4b7e-8d2b-4697dadc4dbc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" Aug 04 16:02:12.441030 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440601 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b557e1a0-4af6-4c29-bb50-214fa9d6882e-config-volume\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:12.441030 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64e539db-01a3-48e9-bf35-35807cebc38f-trusted-ca\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.441030 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440689 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64e539db-01a3-48e9-bf35-35807cebc38f-installation-pull-secrets\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.441030 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.440803 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb89b27f-9d1b-41b0-830e-1e464d29d6f1-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v8nwn\" (UID: \"cb89b27f-9d1b-41b0-830e-1e464d29d6f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn" Aug 04 16:02:12.441657 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.441637 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64e539db-01a3-48e9-bf35-35807cebc38f-trusted-ca\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.443366 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.443349 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb89b27f-9d1b-41b0-830e-1e464d29d6f1-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v8nwn\" (UID: \"cb89b27f-9d1b-41b0-830e-1e464d29d6f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn" Aug 04 16:02:12.443528 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.443512 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64e539db-01a3-48e9-bf35-35807cebc38f-installation-pull-secrets\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.443586 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.443575 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64e539db-01a3-48e9-bf35-35807cebc38f-image-registry-private-configuration\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.450147 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.450119 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-bound-sa-token\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.450492 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.450462 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvrks\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-kube-api-access-hvrks\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.450672 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.450656 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn856\" (UniqueName: \"kubernetes.io/projected/cb89b27f-9d1b-41b0-830e-1e464d29d6f1-kube-api-access-mn856\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v8nwn\" (UID: \"cb89b27f-9d1b-41b0-830e-1e464d29d6f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn" Aug 04 16:02:12.474120 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.474098 2580 generic.go:356] "Generic (PLEG): container finished" podID="a9c0f306-ee82-4a4a-9680-5b0835964025" containerID="002458ae83e1d909dca07512ec18e6df0161d7e7e6472e8ec67c796b2dc20920" exitCode=0 Aug 04 16:02:12.474296 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.474144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sdq7" event={"ID":"a9c0f306-ee82-4a4a-9680-5b0835964025","Type":"ContainerDied","Data":"002458ae83e1d909dca07512ec18e6df0161d7e7e6472e8ec67c796b2dc20920"} Aug 04 16:02:12.541372 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-default-certificate\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:12.541504 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541377 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:12.541504 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b557e1a0-4af6-4c29-bb50-214fa9d6882e-tmp-dir\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:12.541504 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541408 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-rz5dr\" (UID: \"84092957-5deb-4cae-b1ba-310f24137f8e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:12.541504 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541425 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmfv9\" (UniqueName: \"kubernetes.io/projected/01d084ac-ce47-4d96-bbf0-2e78b83beb17-kube-api-access-nmfv9\") pod \"console-operator-5ccd6b445f-jcbjw\" (UID: \"01d084ac-ce47-4d96-bbf0-2e78b83beb17\") " pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:12.541504 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541441 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tcsrs\" (UID: \"6f235ea2-00aa-471c-b792-a2ed764869fa\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" Aug 04 16:02:12.541504 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541461 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb6p7\" (UniqueName: \"kubernetes.io/projected/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-kube-api-access-wb6p7\") pod \"cluster-samples-operator-559cbf5745-n4zpv\" (UID: \"e9647849-9d1f-4b7e-8d2b-4697dadc4dbc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" Aug 04 16:02:12.541504 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.541495 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 16:02:12.541504 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.541498 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 16:02:12.541877 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b557e1a0-4af6-4c29-bb50-214fa9d6882e-config-volume\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:12.541877 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.541568 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls podName:b557e1a0-4af6-4c29-bb50-214fa9d6882e nodeName:}" failed. No retries permitted until 2026-08-04 16:02:13.041547147 +0000 UTC m=+34.185879372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls") pod "dns-default-zj6tj" (UID: "b557e1a0-4af6-4c29-bb50-214fa9d6882e") : secret "dns-default-metrics-tls" not found Aug 04 16:02:12.541877 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541606 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:12.541877 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.541619 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls podName:84092957-5deb-4cae-b1ba-310f24137f8e nodeName:}" failed. No retries permitted until 2026-08-04 16:02:13.041608691 +0000 UTC m=+34.185940916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-rz5dr" (UID: "84092957-5deb-4cae-b1ba-310f24137f8e") : secret "cluster-monitoring-operator-tls" not found Aug 04 16:02:12.541877 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln589\" (UniqueName: \"kubernetes.io/projected/687ce5a4-b2e9-4c41-bb89-89d423b1e514-kube-api-access-ln589\") pod \"service-ca-operator-784f97bcf6-sg8sp\" (UID: \"687ce5a4-b2e9-4c41-bb89-89d423b1e514\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp" Aug 04 16:02:12.541877 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541701 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b557e1a0-4af6-4c29-bb50-214fa9d6882e-tmp-dir\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:12.541877 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-n4zpv\" (UID: \"e9647849-9d1f-4b7e-8d2b-4697dadc4dbc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" Aug 04 16:02:12.541877 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541744 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01d084ac-ce47-4d96-bbf0-2e78b83beb17-serving-cert\") pod \"console-operator-5ccd6b445f-jcbjw\" (UID: \"01d084ac-ce47-4d96-bbf0-2e78b83beb17\") " pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:12.541877 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.541758 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 16:02:12.541877 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:12.541877 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541791 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq672\" (UniqueName: \"kubernetes.io/projected/a9b9081e-ce41-4a27-97c9-d02d7bd2bab7-kube-api-access-fq672\") pod \"ingress-canary-v8jrh\" (UID: \"a9b9081e-ce41-4a27-97c9-d02d7bd2bab7\") " pod="openshift-ingress-canary/ingress-canary-v8jrh" Aug 04 16:02:12.541877 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.541814 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls podName:e9647849-9d1f-4b7e-8d2b-4697dadc4dbc nodeName:}" failed. No retries permitted until 2026-08-04 16:02:13.041798294 +0000 UTC m=+34.186130521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-n4zpv" (UID: "e9647849-9d1f-4b7e-8d2b-4697dadc4dbc") : secret "samples-operator-tls" not found Aug 04 16:02:12.541877 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e015152-d986-4b2b-9f93-296d9291b74e-tmp\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.542517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01d084ac-ce47-4d96-bbf0-2e78b83beb17-config\") pod \"console-operator-5ccd6b445f-jcbjw\" (UID: \"01d084ac-ce47-4d96-bbf0-2e78b83beb17\") " pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:12.542517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541945 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2wtx\" (UniqueName: \"kubernetes.io/projected/66928c11-68ad-4888-8ee8-82a21af92095-kube-api-access-h2wtx\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:12.542517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541971 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/687ce5a4-b2e9-4c41-bb89-89d423b1e514-serving-cert\") pod \"service-ca-operator-784f97bcf6-sg8sp\" (UID: \"687ce5a4-b2e9-4c41-bb89-89d423b1e514\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp" Aug 04 16:02:12.542517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.541999 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6f235ea2-00aa-471c-b792-a2ed764869fa-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-tcsrs\" (UID: \"6f235ea2-00aa-471c-b792-a2ed764869fa\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" Aug 04 16:02:12.542517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e015152-d986-4b2b-9f93-296d9291b74e-trusted-ca-bundle\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.542517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9b9081e-ce41-4a27-97c9-d02d7bd2bab7-cert\") pod \"ingress-canary-v8jrh\" (UID: \"a9b9081e-ce41-4a27-97c9-d02d7bd2bab7\") " pod="openshift-ingress-canary/ingress-canary-v8jrh" Aug 04 16:02:12.542517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542092 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b557e1a0-4af6-4c29-bb50-214fa9d6882e-config-volume\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:12.542517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c72wt\" (UniqueName: \"kubernetes.io/projected/84092957-5deb-4cae-b1ba-310f24137f8e-kube-api-access-c72wt\") pod \"cluster-monitoring-operator-6f55c4f58d-rz5dr\" (UID: \"84092957-5deb-4cae-b1ba-310f24137f8e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:12.542517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542144 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e015152-d986-4b2b-9f93-296d9291b74e-tmp\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.542517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/687ce5a4-b2e9-4c41-bb89-89d423b1e514-config\") pod \"service-ca-operator-784f97bcf6-sg8sp\" (UID: \"687ce5a4-b2e9-4c41-bb89-89d423b1e514\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp" Aug 04 16:02:12.542517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542233 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01d084ac-ce47-4d96-bbf0-2e78b83beb17-trusted-ca\") pod \"console-operator-5ccd6b445f-jcbjw\" (UID: \"01d084ac-ce47-4d96-bbf0-2e78b83beb17\") " pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:12.542517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542258 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e015152-d986-4b2b-9f93-296d9291b74e-serving-cert\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.542517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/84092957-5deb-4cae-b1ba-310f24137f8e-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-rz5dr\" (UID: \"84092957-5deb-4cae-b1ba-310f24137f8e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:12.542517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542323 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-stats-auth\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:12.542517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542355 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7sf7\" (UniqueName: \"kubernetes.io/projected/cdaafd1b-8147-4e4d-82c5-578621b9a646-kube-api-access-q7sf7\") pod \"volume-data-source-validator-5b8776669-7fvfv\" (UID: \"cdaafd1b-8147-4e4d-82c5-578621b9a646\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-7fvfv" Aug 04 16:02:12.542517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542391 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8s44\" (UniqueName: \"kubernetes.io/projected/ff0c52c1-71f1-42ec-89d2-4ef28d8aa1cb-kube-api-access-g8s44\") pod \"network-check-source-65bdbbff45-krplv\" (UID: \"ff0c52c1-71f1-42ec-89d2-4ef28d8aa1cb\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-krplv" Aug 04 16:02:12.543254 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwd2p\" (UniqueName: \"kubernetes.io/projected/b557e1a0-4af6-4c29-bb50-214fa9d6882e-kube-api-access-cwd2p\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:12.543254 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e015152-d986-4b2b-9f93-296d9291b74e-service-ca-bundle\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.543254 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwv4\" (UniqueName: \"kubernetes.io/projected/8e015152-d986-4b2b-9f93-296d9291b74e-kube-api-access-qgwv4\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.543254 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8e015152-d986-4b2b-9f93-296d9291b74e-snapshots\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.543254 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542560 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01d084ac-ce47-4d96-bbf0-2e78b83beb17-config\") pod \"console-operator-5ccd6b445f-jcbjw\" (UID: \"01d084ac-ce47-4d96-bbf0-2e78b83beb17\") " pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:12.543254 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542755 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/687ce5a4-b2e9-4c41-bb89-89d423b1e514-config\") pod \"service-ca-operator-784f97bcf6-sg8sp\" (UID: \"687ce5a4-b2e9-4c41-bb89-89d423b1e514\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp" Aug 04 16:02:12.543254 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.542899 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e015152-d986-4b2b-9f93-296d9291b74e-trusted-ca-bundle\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.543254 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.543149 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e015152-d986-4b2b-9f93-296d9291b74e-service-ca-bundle\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.543254 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.543152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01d084ac-ce47-4d96-bbf0-2e78b83beb17-trusted-ca\") pod \"console-operator-5ccd6b445f-jcbjw\" (UID: \"01d084ac-ce47-4d96-bbf0-2e78b83beb17\") " pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:12.543697 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.543403 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8e015152-d986-4b2b-9f93-296d9291b74e-snapshots\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.543697 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.543532 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/84092957-5deb-4cae-b1ba-310f24137f8e-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-rz5dr\" (UID: \"84092957-5deb-4cae-b1ba-310f24137f8e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:12.544367 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.544349 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/687ce5a4-b2e9-4c41-bb89-89d423b1e514-serving-cert\") pod \"service-ca-operator-784f97bcf6-sg8sp\" (UID: \"687ce5a4-b2e9-4c41-bb89-89d423b1e514\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp" Aug 04 16:02:12.544673 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.544655 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e015152-d986-4b2b-9f93-296d9291b74e-serving-cert\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.544804 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.544790 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01d084ac-ce47-4d96-bbf0-2e78b83beb17-serving-cert\") pod \"console-operator-5ccd6b445f-jcbjw\" (UID: \"01d084ac-ce47-4d96-bbf0-2e78b83beb17\") " pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:12.550630 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.550607 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c72wt\" (UniqueName: \"kubernetes.io/projected/84092957-5deb-4cae-b1ba-310f24137f8e-kube-api-access-c72wt\") pod \"cluster-monitoring-operator-6f55c4f58d-rz5dr\" (UID: \"84092957-5deb-4cae-b1ba-310f24137f8e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:12.550903 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.550881 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln589\" (UniqueName: \"kubernetes.io/projected/687ce5a4-b2e9-4c41-bb89-89d423b1e514-kube-api-access-ln589\") pod \"service-ca-operator-784f97bcf6-sg8sp\" (UID: \"687ce5a4-b2e9-4c41-bb89-89d423b1e514\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp" Aug 04 16:02:12.551035 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.550888 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb6p7\" (UniqueName: \"kubernetes.io/projected/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-kube-api-access-wb6p7\") pod \"cluster-samples-operator-559cbf5745-n4zpv\" (UID: \"e9647849-9d1f-4b7e-8d2b-4697dadc4dbc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" Aug 04 16:02:12.551120 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.551105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmfv9\" (UniqueName: \"kubernetes.io/projected/01d084ac-ce47-4d96-bbf0-2e78b83beb17-kube-api-access-nmfv9\") pod \"console-operator-5ccd6b445f-jcbjw\" (UID: \"01d084ac-ce47-4d96-bbf0-2e78b83beb17\") " pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:12.551493 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.551470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwv4\" (UniqueName: \"kubernetes.io/projected/8e015152-d986-4b2b-9f93-296d9291b74e-kube-api-access-qgwv4\") pod \"insights-operator-77696685b-8scs4\" (UID: \"8e015152-d986-4b2b-9f93-296d9291b74e\") " pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.551571 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.551555 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwd2p\" (UniqueName: \"kubernetes.io/projected/b557e1a0-4af6-4c29-bb50-214fa9d6882e-kube-api-access-cwd2p\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:12.551617 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.551579 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7sf7\" (UniqueName: \"kubernetes.io/projected/cdaafd1b-8147-4e4d-82c5-578621b9a646-kube-api-access-q7sf7\") pod \"volume-data-source-validator-5b8776669-7fvfv\" (UID: \"cdaafd1b-8147-4e4d-82c5-578621b9a646\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-7fvfv" Aug 04 16:02:12.557115 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.557097 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn" Aug 04 16:02:12.614497 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.614459 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:12.627680 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.627660 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp" Aug 04 16:02:12.636829 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.636803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-7fvfv" Aug 04 16:02:12.643346 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.643316 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-default-certificate\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:12.643557 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.643402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tcsrs\" (UID: \"6f235ea2-00aa-471c-b792-a2ed764869fa\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" Aug 04 16:02:12.643642 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.643599 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 16:02:12.643693 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.643664 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert podName:6f235ea2-00aa-471c-b792-a2ed764869fa nodeName:}" failed. No retries permitted until 2026-08-04 16:02:13.143642283 +0000 UTC m=+34.287974513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-tcsrs" (UID: "6f235ea2-00aa-471c-b792-a2ed764869fa") : secret "networking-console-plugin-cert" not found Aug 04 16:02:12.644106 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.644078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:12.644199 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.644159 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:12.644246 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.644194 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq672\" (UniqueName: \"kubernetes.io/projected/a9b9081e-ce41-4a27-97c9-d02d7bd2bab7-kube-api-access-fq672\") pod \"ingress-canary-v8jrh\" (UID: \"a9b9081e-ce41-4a27-97c9-d02d7bd2bab7\") " pod="openshift-ingress-canary/ingress-canary-v8jrh" Aug 04 16:02:12.644319 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.644242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2wtx\" (UniqueName: \"kubernetes.io/projected/66928c11-68ad-4888-8ee8-82a21af92095-kube-api-access-h2wtx\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:12.644319 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.644272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6f235ea2-00aa-471c-b792-a2ed764869fa-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-tcsrs\" (UID: \"6f235ea2-00aa-471c-b792-a2ed764869fa\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" Aug 04 16:02:12.644414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.644325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9b9081e-ce41-4a27-97c9-d02d7bd2bab7-cert\") pod \"ingress-canary-v8jrh\" (UID: \"a9b9081e-ce41-4a27-97c9-d02d7bd2bab7\") " pod="openshift-ingress-canary/ingress-canary-v8jrh" Aug 04 16:02:12.644414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.644363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-stats-auth\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:12.644414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.644402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8s44\" (UniqueName: \"kubernetes.io/projected/ff0c52c1-71f1-42ec-89d2-4ef28d8aa1cb-kube-api-access-g8s44\") pod \"network-check-source-65bdbbff45-krplv\" (UID: \"ff0c52c1-71f1-42ec-89d2-4ef28d8aa1cb\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-krplv" Aug 04 16:02:12.645900 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.645731 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle podName:66928c11-68ad-4888-8ee8-82a21af92095 nodeName:}" failed. No retries permitted until 2026-08-04 16:02:13.145713062 +0000 UTC m=+34.290045300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle") pod "router-default-6d8f46644-nmgqm" (UID: "66928c11-68ad-4888-8ee8-82a21af92095") : configmap references non-existent config key: service-ca.crt Aug 04 16:02:12.645900 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.645785 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 16:02:12.645900 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.645849 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs podName:66928c11-68ad-4888-8ee8-82a21af92095 nodeName:}" failed. No retries permitted until 2026-08-04 16:02:13.145833111 +0000 UTC m=+34.290165340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs") pod "router-default-6d8f46644-nmgqm" (UID: "66928c11-68ad-4888-8ee8-82a21af92095") : secret "router-metrics-certs-default" not found Aug 04 16:02:12.646643 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.646620 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6f235ea2-00aa-471c-b792-a2ed764869fa-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-tcsrs\" (UID: \"6f235ea2-00aa-471c-b792-a2ed764869fa\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" Aug 04 16:02:12.648687 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.648655 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-stats-auth\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:12.649402 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.649100 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9b9081e-ce41-4a27-97c9-d02d7bd2bab7-cert\") pod \"ingress-canary-v8jrh\" (UID: \"a9b9081e-ce41-4a27-97c9-d02d7bd2bab7\") " pod="openshift-ingress-canary/ingress-canary-v8jrh" Aug 04 16:02:12.650009 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.649984 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-default-certificate\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:12.658363 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.657922 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8s44\" (UniqueName: \"kubernetes.io/projected/ff0c52c1-71f1-42ec-89d2-4ef28d8aa1cb-kube-api-access-g8s44\") pod \"network-check-source-65bdbbff45-krplv\" (UID: \"ff0c52c1-71f1-42ec-89d2-4ef28d8aa1cb\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-krplv" Aug 04 16:02:12.660400 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.660381 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-8scs4" Aug 04 16:02:12.660802 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.660777 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq672\" (UniqueName: \"kubernetes.io/projected/a9b9081e-ce41-4a27-97c9-d02d7bd2bab7-kube-api-access-fq672\") pod \"ingress-canary-v8jrh\" (UID: \"a9b9081e-ce41-4a27-97c9-d02d7bd2bab7\") " pod="openshift-ingress-canary/ingress-canary-v8jrh" Aug 04 16:02:12.661834 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.661793 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2wtx\" (UniqueName: \"kubernetes.io/projected/66928c11-68ad-4888-8ee8-82a21af92095-kube-api-access-h2wtx\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:12.704674 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.704184 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v8jrh" Aug 04 16:02:12.719706 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.719609 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-krplv" Aug 04 16:02:12.741199 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.741037 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn"] Aug 04 16:02:12.752002 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:02:12.751620 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb89b27f_9d1b_41b0_830e_1e464d29d6f1.slice/crio-c4099129df5239a5451cc3e6ba6ebb9a0d951f3e975837565a4230ffc60e2796 WatchSource:0}: Error finding container c4099129df5239a5451cc3e6ba6ebb9a0d951f3e975837565a4230ffc60e2796: Status 404 returned error can't find the container with id c4099129df5239a5451cc3e6ba6ebb9a0d951f3e975837565a4230ffc60e2796 Aug 04 16:02:12.799922 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.799346 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-jcbjw"] Aug 04 16:02:12.805376 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:02:12.804074 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d084ac_ce47_4d96_bbf0_2e78b83beb17.slice/crio-93d315fcb256cdc10ac6b1f0a1e76b33903ea8dc7c25d1fb7f2e7b19a2f945bc WatchSource:0}: Error finding container 93d315fcb256cdc10ac6b1f0a1e76b33903ea8dc7c25d1fb7f2e7b19a2f945bc: Status 404 returned error can't find the container with id 93d315fcb256cdc10ac6b1f0a1e76b33903ea8dc7c25d1fb7f2e7b19a2f945bc Aug 04 16:02:12.828872 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.828700 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp"] Aug 04 16:02:12.832779 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:02:12.832733 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687ce5a4_b2e9_4c41_bb89_89d423b1e514.slice/crio-2f239d4436dc57af7486b9cff5b31bfc41afff104e4197e40e4d769d9fca3f15 WatchSource:0}: Error finding container 2f239d4436dc57af7486b9cff5b31bfc41afff104e4197e40e4d769d9fca3f15: Status 404 returned error can't find the container with id 2f239d4436dc57af7486b9cff5b31bfc41afff104e4197e40e4d769d9fca3f15 Aug 04 16:02:12.866324 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.864094 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-7fvfv"] Aug 04 16:02:12.895402 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.895376 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-8scs4"] Aug 04 16:02:12.899891 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:02:12.899864 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e015152_d986_4b2b_9f93_296d9291b74e.slice/crio-4ce5cfb2bdee374472611042006531b04f7c8b9b71f8733ddebbaee7b6c3b525 WatchSource:0}: Error finding container 4ce5cfb2bdee374472611042006531b04f7c8b9b71f8733ddebbaee7b6c3b525: Status 404 returned error can't find the container with id 4ce5cfb2bdee374472611042006531b04f7c8b9b71f8733ddebbaee7b6c3b525 Aug 04 16:02:12.909334 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.909311 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v8jrh"] Aug 04 16:02:12.912265 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:02:12.912243 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b9081e_ce41_4a27_97c9_d02d7bd2bab7.slice/crio-a6242a16798d94216044294e2bea951f1fdf27cdacf2ada8a79d5d75a25e47bc WatchSource:0}: Error finding container a6242a16798d94216044294e2bea951f1fdf27cdacf2ada8a79d5d75a25e47bc: Status 404 returned error can't find the container with id a6242a16798d94216044294e2bea951f1fdf27cdacf2ada8a79d5d75a25e47bc Aug 04 16:02:12.936671 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.936652 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-krplv"] Aug 04 16:02:12.939325 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:02:12.939305 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0c52c1_71f1_42ec_89d2_4ef28d8aa1cb.slice/crio-52e15febd7e4b3608773ef7e7f15e873b0e1ed317b9b04201cb6baea0d467d90 WatchSource:0}: Error finding container 52e15febd7e4b3608773ef7e7f15e873b0e1ed317b9b04201cb6baea0d467d90: Status 404 returned error can't find the container with id 52e15febd7e4b3608773ef7e7f15e873b0e1ed317b9b04201cb6baea0d467d90 Aug 04 16:02:12.951768 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:12.951703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:12.951858 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.951842 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 16:02:12.951896 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.951859 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p: secret "image-registry-tls" not found Aug 04 16:02:12.951931 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:12.951902 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls podName:64e539db-01a3-48e9-bf35-35807cebc38f nodeName:}" failed. No retries permitted until 2026-08-04 16:02:13.951888474 +0000 UTC m=+35.096220698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls") pod "image-registry-7c7ddb6d8b-g9s7p" (UID: "64e539db-01a3-48e9-bf35-35807cebc38f") : secret "image-registry-tls" not found Aug 04 16:02:13.053899 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:13.053407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:13.054038 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:13.053918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-rz5dr\" (UID: \"84092957-5deb-4cae-b1ba-310f24137f8e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:13.054102 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:13.054034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-n4zpv\" (UID: \"e9647849-9d1f-4b7e-8d2b-4697dadc4dbc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" Aug 04 16:02:13.056400 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:13.054266 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 16:02:13.056400 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:13.054346 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls podName:e9647849-9d1f-4b7e-8d2b-4697dadc4dbc nodeName:}" failed. No retries permitted until 2026-08-04 16:02:14.054328541 +0000 UTC m=+35.198660784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-n4zpv" (UID: "e9647849-9d1f-4b7e-8d2b-4697dadc4dbc") : secret "samples-operator-tls" not found Aug 04 16:02:13.056400 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:13.054771 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 16:02:13.056400 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:13.054831 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls podName:b557e1a0-4af6-4c29-bb50-214fa9d6882e nodeName:}" failed. No retries permitted until 2026-08-04 16:02:14.054809595 +0000 UTC m=+35.199141833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls") pod "dns-default-zj6tj" (UID: "b557e1a0-4af6-4c29-bb50-214fa9d6882e") : secret "dns-default-metrics-tls" not found Aug 04 16:02:13.056400 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:13.054889 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 16:02:13.056400 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:13.054926 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls podName:84092957-5deb-4cae-b1ba-310f24137f8e nodeName:}" failed. No retries permitted until 2026-08-04 16:02:14.054916606 +0000 UTC m=+35.199248832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-rz5dr" (UID: "84092957-5deb-4cae-b1ba-310f24137f8e") : secret "cluster-monitoring-operator-tls" not found Aug 04 16:02:13.154889 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:13.154865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tcsrs\" (UID: \"6f235ea2-00aa-471c-b792-a2ed764869fa\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" Aug 04 16:02:13.155041 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:13.154899 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:13.155041 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:13.154938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:13.155041 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:13.155010 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 16:02:13.155041 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:13.155028 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 16:02:13.155205 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:13.155016 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle podName:66928c11-68ad-4888-8ee8-82a21af92095 nodeName:}" failed. No retries permitted until 2026-08-04 16:02:14.155002521 +0000 UTC m=+35.299334746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle") pod "router-default-6d8f46644-nmgqm" (UID: "66928c11-68ad-4888-8ee8-82a21af92095") : configmap references non-existent config key: service-ca.crt Aug 04 16:02:13.155205 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:13.155077 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert podName:6f235ea2-00aa-471c-b792-a2ed764869fa nodeName:}" failed. No retries permitted until 2026-08-04 16:02:14.155060519 +0000 UTC m=+35.299392744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-tcsrs" (UID: "6f235ea2-00aa-471c-b792-a2ed764869fa") : secret "networking-console-plugin-cert" not found Aug 04 16:02:13.155205 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:13.155100 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs podName:66928c11-68ad-4888-8ee8-82a21af92095 nodeName:}" failed. No retries permitted until 2026-08-04 16:02:14.155090417 +0000 UTC m=+35.299422642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs") pod "router-default-6d8f46644-nmgqm" (UID: "66928c11-68ad-4888-8ee8-82a21af92095") : secret "router-metrics-certs-default" not found Aug 04 16:02:13.480076 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:13.479987 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-7fvfv" event={"ID":"cdaafd1b-8147-4e4d-82c5-578621b9a646","Type":"ContainerStarted","Data":"f4d4aebb4a5671a12e81deecd6f4e24c211fe7cd6f6524beb2e642477a0a5c0b"} Aug 04 16:02:13.483820 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:13.483759 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v8jrh" event={"ID":"a9b9081e-ce41-4a27-97c9-d02d7bd2bab7","Type":"ContainerStarted","Data":"a6242a16798d94216044294e2bea951f1fdf27cdacf2ada8a79d5d75a25e47bc"} Aug 04 16:02:13.485865 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:13.485817 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-8scs4" event={"ID":"8e015152-d986-4b2b-9f93-296d9291b74e","Type":"ContainerStarted","Data":"4ce5cfb2bdee374472611042006531b04f7c8b9b71f8733ddebbaee7b6c3b525"} Aug 04 16:02:13.487933 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:13.487889 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp" event={"ID":"687ce5a4-b2e9-4c41-bb89-89d423b1e514","Type":"ContainerStarted","Data":"2f239d4436dc57af7486b9cff5b31bfc41afff104e4197e40e4d769d9fca3f15"} Aug 04 16:02:13.489188 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:13.489138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" event={"ID":"01d084ac-ce47-4d96-bbf0-2e78b83beb17","Type":"ContainerStarted","Data":"93d315fcb256cdc10ac6b1f0a1e76b33903ea8dc7c25d1fb7f2e7b19a2f945bc"} Aug 04 16:02:13.490829 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:13.490808 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-krplv" event={"ID":"ff0c52c1-71f1-42ec-89d2-4ef28d8aa1cb","Type":"ContainerStarted","Data":"52e15febd7e4b3608773ef7e7f15e873b0e1ed317b9b04201cb6baea0d467d90"} Aug 04 16:02:13.492240 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:13.492193 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn" event={"ID":"cb89b27f-9d1b-41b0-830e-1e464d29d6f1","Type":"ContainerStarted","Data":"c4099129df5239a5451cc3e6ba6ebb9a0d951f3e975837565a4230ffc60e2796"} Aug 04 16:02:13.962091 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:13.962066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:13.962469 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:13.962221 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 16:02:13.962469 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:13.962239 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p: secret "image-registry-tls" not found Aug 04 16:02:13.962469 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:13.962307 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls podName:64e539db-01a3-48e9-bf35-35807cebc38f nodeName:}" failed. No retries permitted until 2026-08-04 16:02:15.962275454 +0000 UTC m=+37.106607679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls") pod "image-registry-7c7ddb6d8b-g9s7p" (UID: "64e539db-01a3-48e9-bf35-35807cebc38f") : secret "image-registry-tls" not found Aug 04 16:02:14.063156 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:14.063131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:14.063291 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:14.063172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-rz5dr\" (UID: \"84092957-5deb-4cae-b1ba-310f24137f8e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:14.063291 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:14.063272 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 16:02:14.063425 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:14.063330 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-n4zpv\" (UID: \"e9647849-9d1f-4b7e-8d2b-4697dadc4dbc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" Aug 04 16:02:14.063425 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:14.063344 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls podName:84092957-5deb-4cae-b1ba-310f24137f8e nodeName:}" failed. No retries permitted until 2026-08-04 16:02:16.063325291 +0000 UTC m=+37.207657520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-rz5dr" (UID: "84092957-5deb-4cae-b1ba-310f24137f8e") : secret "cluster-monitoring-operator-tls" not found Aug 04 16:02:14.063425 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:14.063364 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 16:02:14.063425 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:14.063413 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls podName:b557e1a0-4af6-4c29-bb50-214fa9d6882e nodeName:}" failed. No retries permitted until 2026-08-04 16:02:16.063398131 +0000 UTC m=+37.207730355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls") pod "dns-default-zj6tj" (UID: "b557e1a0-4af6-4c29-bb50-214fa9d6882e") : secret "dns-default-metrics-tls" not found Aug 04 16:02:14.063615 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:14.063457 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 16:02:14.063615 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:14.063517 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls podName:e9647849-9d1f-4b7e-8d2b-4697dadc4dbc nodeName:}" failed. No retries permitted until 2026-08-04 16:02:16.063501063 +0000 UTC m=+37.207833299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-n4zpv" (UID: "e9647849-9d1f-4b7e-8d2b-4697dadc4dbc") : secret "samples-operator-tls" not found Aug 04 16:02:14.164057 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:14.164026 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tcsrs\" (UID: \"6f235ea2-00aa-471c-b792-a2ed764869fa\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" Aug 04 16:02:14.164194 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:14.164066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:14.164254 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:14.164193 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 16:02:14.164329 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:14.164201 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:14.164329 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:14.164267 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert podName:6f235ea2-00aa-471c-b792-a2ed764869fa nodeName:}" failed. No retries permitted until 2026-08-04 16:02:16.164246107 +0000 UTC m=+37.308578349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-tcsrs" (UID: "6f235ea2-00aa-471c-b792-a2ed764869fa") : secret "networking-console-plugin-cert" not found Aug 04 16:02:14.164329 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:14.164270 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 16:02:14.164329 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:14.164313 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle podName:66928c11-68ad-4888-8ee8-82a21af92095 nodeName:}" failed. No retries permitted until 2026-08-04 16:02:16.164300634 +0000 UTC m=+37.308632862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle") pod "router-default-6d8f46644-nmgqm" (UID: "66928c11-68ad-4888-8ee8-82a21af92095") : configmap references non-existent config key: service-ca.crt Aug 04 16:02:14.164471 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:14.164342 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs podName:66928c11-68ad-4888-8ee8-82a21af92095 nodeName:}" failed. No retries permitted until 2026-08-04 16:02:16.164329494 +0000 UTC m=+37.308661721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs") pod "router-default-6d8f46644-nmgqm" (UID: "66928c11-68ad-4888-8ee8-82a21af92095") : secret "router-metrics-certs-default" not found Aug 04 16:02:14.497402 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:14.497368 2580 generic.go:356] "Generic (PLEG): container finished" podID="a9c0f306-ee82-4a4a-9680-5b0835964025" containerID="f37e4dbdbcefc7d1f1330964c6b6844861b523f185643c55a98a1889a1d7004c" exitCode=0 Aug 04 16:02:14.497568 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:14.497444 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sdq7" event={"ID":"a9c0f306-ee82-4a4a-9680-5b0835964025","Type":"ContainerDied","Data":"f37e4dbdbcefc7d1f1330964c6b6844861b523f185643c55a98a1889a1d7004c"} Aug 04 16:02:15.982492 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:15.982238 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:15.982943 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:15.982625 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 16:02:15.982943 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:15.982643 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p: secret "image-registry-tls" not found Aug 04 16:02:15.982943 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:15.982700 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls podName:64e539db-01a3-48e9-bf35-35807cebc38f nodeName:}" failed. No retries permitted until 2026-08-04 16:02:19.982681992 +0000 UTC m=+41.127014237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls") pod "image-registry-7c7ddb6d8b-g9s7p" (UID: "64e539db-01a3-48e9-bf35-35807cebc38f") : secret "image-registry-tls" not found Aug 04 16:02:16.083129 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:16.083098 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:16.083306 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:16.083150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-rz5dr\" (UID: \"84092957-5deb-4cae-b1ba-310f24137f8e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:16.083306 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:16.083230 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-n4zpv\" (UID: \"e9647849-9d1f-4b7e-8d2b-4697dadc4dbc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" Aug 04 16:02:16.083465 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:16.083447 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 16:02:16.083531 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:16.083518 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls podName:e9647849-9d1f-4b7e-8d2b-4697dadc4dbc nodeName:}" failed. No retries permitted until 2026-08-04 16:02:20.083498544 +0000 UTC m=+41.227830776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-n4zpv" (UID: "e9647849-9d1f-4b7e-8d2b-4697dadc4dbc") : secret "samples-operator-tls" not found Aug 04 16:02:16.083932 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:16.083908 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 16:02:16.084001 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:16.083967 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls podName:b557e1a0-4af6-4c29-bb50-214fa9d6882e nodeName:}" failed. No retries permitted until 2026-08-04 16:02:20.08395273 +0000 UTC m=+41.228284957 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls") pod "dns-default-zj6tj" (UID: "b557e1a0-4af6-4c29-bb50-214fa9d6882e") : secret "dns-default-metrics-tls" not found Aug 04 16:02:16.084058 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:16.084018 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 16:02:16.084058 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:16.084054 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls podName:84092957-5deb-4cae-b1ba-310f24137f8e nodeName:}" failed. No retries permitted until 2026-08-04 16:02:20.084041875 +0000 UTC m=+41.228374099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-rz5dr" (UID: "84092957-5deb-4cae-b1ba-310f24137f8e") : secret "cluster-monitoring-operator-tls" not found Aug 04 16:02:16.184207 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:16.184171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tcsrs\" (UID: \"6f235ea2-00aa-471c-b792-a2ed764869fa\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" Aug 04 16:02:16.184386 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:16.184350 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 16:02:16.184450 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:16.184417 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert podName:6f235ea2-00aa-471c-b792-a2ed764869fa nodeName:}" failed. No retries permitted until 2026-08-04 16:02:20.184398248 +0000 UTC m=+41.328730472 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-tcsrs" (UID: "6f235ea2-00aa-471c-b792-a2ed764869fa") : secret "networking-console-plugin-cert" not found Aug 04 16:02:16.185102 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:16.184232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:16.185102 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:16.184795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:16.185102 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:16.184924 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 16:02:16.185102 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:16.184967 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs podName:66928c11-68ad-4888-8ee8-82a21af92095 nodeName:}" failed. No retries permitted until 2026-08-04 16:02:20.184953034 +0000 UTC m=+41.329285278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs") pod "router-default-6d8f46644-nmgqm" (UID: "66928c11-68ad-4888-8ee8-82a21af92095") : secret "router-metrics-certs-default" not found Aug 04 16:02:16.185102 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:16.185072 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle podName:66928c11-68ad-4888-8ee8-82a21af92095 nodeName:}" failed. No retries permitted until 2026-08-04 16:02:20.185061637 +0000 UTC m=+41.329393867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle") pod "router-default-6d8f46644-nmgqm" (UID: "66928c11-68ad-4888-8ee8-82a21af92095") : configmap references non-existent config key: service-ca.crt Aug 04 16:02:16.511089 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:16.509922 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp" event={"ID":"687ce5a4-b2e9-4c41-bb89-89d423b1e514","Type":"ContainerStarted","Data":"9cb23de8d0ef8a37fbdb130eb28c4789cfe9c844dc7f6cadfe80c132f258310d"} Aug 04 16:02:16.518476 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:16.515467 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn" event={"ID":"cb89b27f-9d1b-41b0-830e-1e464d29d6f1","Type":"ContainerStarted","Data":"b178d7a652b408c67d8346a9d89b38e8888f945a93cdf5abf8175d01b858f113"} Aug 04 16:02:16.529835 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:16.528501 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp" podStartSLOduration=18.584798769 podStartE2EDuration="21.528484857s" totalImagesPullingTime="2.943686088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:16.528074831 +0000 UTC m=+37.672407099" watchObservedRunningTime="2026-08-04 16:02:16.528484857 +0000 UTC m=+37.672817105" Aug 04 16:02:18.395172 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:18.395058 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn" podStartSLOduration=20.390838232 podStartE2EDuration="23.395040806s" totalImagesPullingTime="3.004202574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:16.549475633 +0000 UTC m=+37.693807896" watchObservedRunningTime="2026-08-04 16:02:18.395040806 +0000 UTC m=+39.539373051" Aug 04 16:02:18.396107 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:18.396085 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-d8g5c"] Aug 04 16:02:18.422213 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:18.422184 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-d8g5c"] Aug 04 16:02:18.422364 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:18.422336 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d8g5c" Aug 04 16:02:18.425361 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:18.425334 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Aug 04 16:02:18.426543 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:18.426349 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bmgrc\"" Aug 04 16:02:18.426543 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:18.426387 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Aug 04 16:02:18.506541 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:18.506505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnn5d\" (UniqueName: \"kubernetes.io/projected/023c82e6-1623-4606-8fad-5eb6984f70f5-kube-api-access-tnn5d\") pod \"migrator-68bc745d87-d8g5c\" (UID: \"023c82e6-1623-4606-8fad-5eb6984f70f5\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d8g5c" Aug 04 16:02:18.606924 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:18.606891 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnn5d\" (UniqueName: \"kubernetes.io/projected/023c82e6-1623-4606-8fad-5eb6984f70f5-kube-api-access-tnn5d\") pod \"migrator-68bc745d87-d8g5c\" (UID: \"023c82e6-1623-4606-8fad-5eb6984f70f5\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d8g5c" Aug 04 16:02:18.619502 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:18.619471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnn5d\" (UniqueName: \"kubernetes.io/projected/023c82e6-1623-4606-8fad-5eb6984f70f5-kube-api-access-tnn5d\") pod \"migrator-68bc745d87-d8g5c\" (UID: \"023c82e6-1623-4606-8fad-5eb6984f70f5\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d8g5c" Aug 04 16:02:18.733262 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:18.733183 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d8g5c" Aug 04 16:02:19.831037 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:19.831009 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-rmpr4"] Aug 04 16:02:19.853665 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:19.853625 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-rmpr4"] Aug 04 16:02:19.853801 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:19.853688 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-rmpr4" Aug 04 16:02:19.856543 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:19.856512 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Aug 04 16:02:19.856657 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:19.856594 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Aug 04 16:02:19.856736 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:19.856722 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Aug 04 16:02:19.858150 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:19.858008 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nqxm8\"" Aug 04 16:02:19.858150 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:19.858044 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Aug 04 16:02:19.917497 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:19.917470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/825ee096-4758-4b3f-bf58-fee7e0814852-signing-key\") pod \"service-ca-98c88fbb5-rmpr4\" (UID: \"825ee096-4758-4b3f-bf58-fee7e0814852\") " pod="openshift-service-ca/service-ca-98c88fbb5-rmpr4" Aug 04 16:02:19.917633 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:19.917536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89xpm\" (UniqueName: \"kubernetes.io/projected/825ee096-4758-4b3f-bf58-fee7e0814852-kube-api-access-89xpm\") pod \"service-ca-98c88fbb5-rmpr4\" (UID: \"825ee096-4758-4b3f-bf58-fee7e0814852\") " pod="openshift-service-ca/service-ca-98c88fbb5-rmpr4" Aug 04 16:02:19.917633 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:19.917583 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/825ee096-4758-4b3f-bf58-fee7e0814852-signing-cabundle\") pod \"service-ca-98c88fbb5-rmpr4\" (UID: \"825ee096-4758-4b3f-bf58-fee7e0814852\") " pod="openshift-service-ca/service-ca-98c88fbb5-rmpr4" Aug 04 16:02:20.018948 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:20.018915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/825ee096-4758-4b3f-bf58-fee7e0814852-signing-cabundle\") pod \"service-ca-98c88fbb5-rmpr4\" (UID: \"825ee096-4758-4b3f-bf58-fee7e0814852\") " pod="openshift-service-ca/service-ca-98c88fbb5-rmpr4" Aug 04 16:02:20.019111 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:20.019053 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/825ee096-4758-4b3f-bf58-fee7e0814852-signing-key\") pod \"service-ca-98c88fbb5-rmpr4\" (UID: \"825ee096-4758-4b3f-bf58-fee7e0814852\") " pod="openshift-service-ca/service-ca-98c88fbb5-rmpr4" Aug 04 16:02:20.019211 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:20.019194 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89xpm\" (UniqueName: \"kubernetes.io/projected/825ee096-4758-4b3f-bf58-fee7e0814852-kube-api-access-89xpm\") pod \"service-ca-98c88fbb5-rmpr4\" (UID: \"825ee096-4758-4b3f-bf58-fee7e0814852\") " pod="openshift-service-ca/service-ca-98c88fbb5-rmpr4" Aug 04 16:02:20.019276 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:20.019232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:20.019374 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:20.019359 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 16:02:20.019429 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:20.019378 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p: secret "image-registry-tls" not found Aug 04 16:02:20.019429 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:20.019418 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls podName:64e539db-01a3-48e9-bf35-35807cebc38f nodeName:}" failed. No retries permitted until 2026-08-04 16:02:28.019406844 +0000 UTC m=+49.163739069 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls") pod "image-registry-7c7ddb6d8b-g9s7p" (UID: "64e539db-01a3-48e9-bf35-35807cebc38f") : secret "image-registry-tls" not found Aug 04 16:02:20.019703 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:20.019681 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/825ee096-4758-4b3f-bf58-fee7e0814852-signing-cabundle\") pod \"service-ca-98c88fbb5-rmpr4\" (UID: \"825ee096-4758-4b3f-bf58-fee7e0814852\") " pod="openshift-service-ca/service-ca-98c88fbb5-rmpr4" Aug 04 16:02:20.021877 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:20.021854 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/825ee096-4758-4b3f-bf58-fee7e0814852-signing-key\") pod \"service-ca-98c88fbb5-rmpr4\" (UID: \"825ee096-4758-4b3f-bf58-fee7e0814852\") " pod="openshift-service-ca/service-ca-98c88fbb5-rmpr4" Aug 04 16:02:20.029365 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:20.029343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89xpm\" (UniqueName: \"kubernetes.io/projected/825ee096-4758-4b3f-bf58-fee7e0814852-kube-api-access-89xpm\") pod \"service-ca-98c88fbb5-rmpr4\" (UID: \"825ee096-4758-4b3f-bf58-fee7e0814852\") " pod="openshift-service-ca/service-ca-98c88fbb5-rmpr4" Aug 04 16:02:20.120383 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:20.119808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-rz5dr\" (UID: \"84092957-5deb-4cae-b1ba-310f24137f8e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:20.120383 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:20.119911 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-n4zpv\" (UID: \"e9647849-9d1f-4b7e-8d2b-4697dadc4dbc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" Aug 04 16:02:20.120383 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:20.120051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:20.120383 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:20.120179 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 16:02:20.120383 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:20.120199 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 16:02:20.120383 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:20.120200 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 16:02:20.120383 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:20.120261 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls podName:84092957-5deb-4cae-b1ba-310f24137f8e nodeName:}" failed. No retries permitted until 2026-08-04 16:02:28.120242226 +0000 UTC m=+49.264574458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-rz5dr" (UID: "84092957-5deb-4cae-b1ba-310f24137f8e") : secret "cluster-monitoring-operator-tls" not found Aug 04 16:02:20.120383 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:20.120295 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls podName:b557e1a0-4af6-4c29-bb50-214fa9d6882e nodeName:}" failed. No retries permitted until 2026-08-04 16:02:28.120270128 +0000 UTC m=+49.264602352 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls") pod "dns-default-zj6tj" (UID: "b557e1a0-4af6-4c29-bb50-214fa9d6882e") : secret "dns-default-metrics-tls" not found Aug 04 16:02:20.120383 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:20.120309 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls podName:e9647849-9d1f-4b7e-8d2b-4697dadc4dbc nodeName:}" failed. No retries permitted until 2026-08-04 16:02:28.120301689 +0000 UTC m=+49.264633913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-n4zpv" (UID: "e9647849-9d1f-4b7e-8d2b-4697dadc4dbc") : secret "samples-operator-tls" not found Aug 04 16:02:20.164392 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:20.164360 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-rmpr4" Aug 04 16:02:20.221477 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:20.221437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tcsrs\" (UID: \"6f235ea2-00aa-471c-b792-a2ed764869fa\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" Aug 04 16:02:20.221627 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:20.221493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:20.221627 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:20.221537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:20.221627 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:20.221569 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 16:02:20.221782 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:20.221640 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert podName:6f235ea2-00aa-471c-b792-a2ed764869fa nodeName:}" failed. No retries permitted until 2026-08-04 16:02:28.221620105 +0000 UTC m=+49.365952330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-tcsrs" (UID: "6f235ea2-00aa-471c-b792-a2ed764869fa") : secret "networking-console-plugin-cert" not found Aug 04 16:02:20.221782 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:20.221649 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 16:02:20.221782 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:20.221660 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle podName:66928c11-68ad-4888-8ee8-82a21af92095 nodeName:}" failed. No retries permitted until 2026-08-04 16:02:28.221650236 +0000 UTC m=+49.365982460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle") pod "router-default-6d8f46644-nmgqm" (UID: "66928c11-68ad-4888-8ee8-82a21af92095") : configmap references non-existent config key: service-ca.crt Aug 04 16:02:20.221782 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:20.221686 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs podName:66928c11-68ad-4888-8ee8-82a21af92095 nodeName:}" failed. No retries permitted until 2026-08-04 16:02:28.221671818 +0000 UTC m=+49.366004045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs") pod "router-default-6d8f46644-nmgqm" (UID: "66928c11-68ad-4888-8ee8-82a21af92095") : secret "router-metrics-certs-default" not found Aug 04 16:02:21.547536 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:21.547351 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-rmpr4"] Aug 04 16:02:21.566975 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:21.566950 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-d8g5c"] Aug 04 16:02:21.572991 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:02:21.572956 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023c82e6_1623_4606_8fad_5eb6984f70f5.slice/crio-88cb3644af9d8d34c0def440d3ded94cbef0f0382fbd6d5734140f4c438fcef4 WatchSource:0}: Error finding container 88cb3644af9d8d34c0def440d3ded94cbef0f0382fbd6d5734140f4c438fcef4: Status 404 returned error can't find the container with id 88cb3644af9d8d34c0def440d3ded94cbef0f0382fbd6d5734140f4c438fcef4 Aug 04 16:02:22.539766 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.539482 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v8jrh" event={"ID":"a9b9081e-ce41-4a27-97c9-d02d7bd2bab7","Type":"ContainerStarted","Data":"494d140bc06cc27bb77d758a7e173ab2dfa1b446d9b0a707ae8a287520ed309f"} Aug 04 16:02:22.542453 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.541782 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-8scs4" event={"ID":"8e015152-d986-4b2b-9f93-296d9291b74e","Type":"ContainerStarted","Data":"c18b07e9cbb1a425ff9fee6f066d9bc3a609d06e01041f39e26b85df7f86bdea"} Aug 04 16:02:22.543960 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.543614 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" event={"ID":"01d084ac-ce47-4d96-bbf0-2e78b83beb17","Type":"ContainerStarted","Data":"b735854da0e268cae5f060460a73696b62cab6a6fae72bc95b51365fc716202d"} Aug 04 16:02:22.543960 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.543861 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:22.544809 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.544781 2580 patch_prober.go:28] interesting pod/console-operator-5ccd6b445f-jcbjw container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.9:8443/readyz\": dial tcp 10.132.0.9:8443: connect: connection refused" start-of-body= Aug 04 16:02:22.544927 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.544826 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" podUID="01d084ac-ce47-4d96-bbf0-2e78b83beb17" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.9:8443/readyz\": dial tcp 10.132.0.9:8443: connect: connection refused" Aug 04 16:02:22.546380 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.546310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-krplv" event={"ID":"ff0c52c1-71f1-42ec-89d2-4ef28d8aa1cb","Type":"ContainerStarted","Data":"65379b250833be44d688b69f5e229a0ad8e8c608845f0603b3fa0cd3fa411357"} Aug 04 16:02:22.548298 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.548211 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-rmpr4" event={"ID":"825ee096-4758-4b3f-bf58-fee7e0814852","Type":"ContainerStarted","Data":"d2b38451f270fd74ff9c834f9773fe95aca297948022f1f1f8b9c65d7ab54681"} Aug 04 16:02:22.548298 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.548251 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-rmpr4" event={"ID":"825ee096-4758-4b3f-bf58-fee7e0814852","Type":"ContainerStarted","Data":"6cb685819a6a1560d21450aca51901117e84b7e34d6b953738ee474e823271fc"} Aug 04 16:02:22.550326 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.550303 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-7fvfv" event={"ID":"cdaafd1b-8147-4e4d-82c5-578621b9a646","Type":"ContainerStarted","Data":"9d1bf3f1ae628b256f5a81917b6ec304396ded4586710f9f191c9396b237c404"} Aug 04 16:02:22.551445 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.551398 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d8g5c" event={"ID":"023c82e6-1623-4606-8fad-5eb6984f70f5","Type":"ContainerStarted","Data":"88cb3644af9d8d34c0def440d3ded94cbef0f0382fbd6d5734140f4c438fcef4"} Aug 04 16:02:22.556498 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.556453 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v8jrh" podStartSLOduration=2.072136963 podStartE2EDuration="10.55643949s" totalImagesPullingTime="8.484302527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:22.555793613 +0000 UTC m=+43.700125882" watchObservedRunningTime="2026-08-04 16:02:22.55643949 +0000 UTC m=+43.700771734" Aug 04 16:02:22.575706 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.575598 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" podStartSLOduration=19.01510612 podStartE2EDuration="27.575580973s" totalImagesPullingTime="8.560474853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:22.574265851 +0000 UTC m=+43.718598132" watchObservedRunningTime="2026-08-04 16:02:22.575580973 +0000 UTC m=+43.719913221" Aug 04 16:02:22.591254 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.591212 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-7fvfv" podStartSLOduration=19.068738957 podStartE2EDuration="27.591196512s" totalImagesPullingTime="8.522457555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:22.591099116 +0000 UTC m=+43.735431377" watchObservedRunningTime="2026-08-04 16:02:22.591196512 +0000 UTC m=+43.735528752" Aug 04 16:02:22.608588 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.608545 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-8scs4" podStartSLOduration=19.111517328 podStartE2EDuration="27.608531495s" totalImagesPullingTime="8.497014167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:22.608150327 +0000 UTC m=+43.752482600" watchObservedRunningTime="2026-08-04 16:02:22.608531495 +0000 UTC m=+43.752863723" Aug 04 16:02:22.615377 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.615354 2580 patch_prober.go:28] interesting pod/console-operator-5ccd6b445f-jcbjw container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.9:8443/healthz\": dial tcp 10.132.0.9:8443: connect: connection refused" start-of-body= Aug 04 16:02:22.615475 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.615396 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" podUID="01d084ac-ce47-4d96-bbf0-2e78b83beb17" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.9:8443/healthz\": dial tcp 10.132.0.9:8443: connect: connection refused" Aug 04 16:02:22.624573 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.624530 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-krplv" podStartSLOduration=19.156242593 podStartE2EDuration="27.62451556s" totalImagesPullingTime="8.468272967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:22.623921229 +0000 UTC m=+43.768253500" watchObservedRunningTime="2026-08-04 16:02:22.62451556 +0000 UTC m=+43.768847806" Aug 04 16:02:22.639215 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:22.639149 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-rmpr4" podStartSLOduration=3.639135537 podStartE2EDuration="3.639135537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:22.639077835 +0000 UTC m=+43.783410108" watchObservedRunningTime="2026-08-04 16:02:22.639135537 +0000 UTC m=+43.783467783" Aug 04 16:02:23.557333 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:23.556915 2580 patch_prober.go:28] interesting pod/console-operator-5ccd6b445f-jcbjw container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.9:8443/readyz\": dial tcp 10.132.0.9:8443: connect: connection refused" start-of-body= Aug 04 16:02:23.557333 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:23.556972 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" podUID="01d084ac-ce47-4d96-bbf0-2e78b83beb17" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.9:8443/readyz\": dial tcp 10.132.0.9:8443: connect: connection refused" Aug 04 16:02:24.562294 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:24.562116 2580 generic.go:356] "Generic (PLEG): container finished" podID="a9c0f306-ee82-4a4a-9680-5b0835964025" containerID="073f7528ca3107415c393434bcb39fa69a57e2da651bf6ede0e6409b80e9242b" exitCode=0 Aug 04 16:02:24.562294 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:24.562209 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sdq7" event={"ID":"a9c0f306-ee82-4a4a-9680-5b0835964025","Type":"ContainerDied","Data":"073f7528ca3107415c393434bcb39fa69a57e2da651bf6ede0e6409b80e9242b"} Aug 04 16:02:24.563916 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:24.563896 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/0.log" Aug 04 16:02:24.564042 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:24.563931 2580 generic.go:356] "Generic (PLEG): container finished" podID="01d084ac-ce47-4d96-bbf0-2e78b83beb17" containerID="b735854da0e268cae5f060460a73696b62cab6a6fae72bc95b51365fc716202d" exitCode=255 Aug 04 16:02:24.564042 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:24.563979 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" event={"ID":"01d084ac-ce47-4d96-bbf0-2e78b83beb17","Type":"ContainerDied","Data":"b735854da0e268cae5f060460a73696b62cab6a6fae72bc95b51365fc716202d"} Aug 04 16:02:24.564315 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:24.564299 2580 scope.go:117] "RemoveContainer" containerID="b735854da0e268cae5f060460a73696b62cab6a6fae72bc95b51365fc716202d" Aug 04 16:02:25.569594 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:25.569497 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d8g5c" event={"ID":"023c82e6-1623-4606-8fad-5eb6984f70f5","Type":"ContainerStarted","Data":"6131ca55ffa9c47a9228e3a188b44738b2aa08f7f1ec2964c7c002dd5ab8d011"} Aug 04 16:02:25.569995 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:25.569606 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d8g5c" event={"ID":"023c82e6-1623-4606-8fad-5eb6984f70f5","Type":"ContainerStarted","Data":"d0af0fb9991d2f701b828efc6b4acd7b6a2dbda0cb05c9e62398cc2ad30601d4"} Aug 04 16:02:25.572676 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:25.572647 2580 generic.go:356] "Generic (PLEG): container finished" podID="a9c0f306-ee82-4a4a-9680-5b0835964025" containerID="480d7778971185b4c3c2a79cfceb551b8e637a35e16faf82f183712226d0f28c" exitCode=0 Aug 04 16:02:25.572783 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:25.572727 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sdq7" event={"ID":"a9c0f306-ee82-4a4a-9680-5b0835964025","Type":"ContainerDied","Data":"480d7778971185b4c3c2a79cfceb551b8e637a35e16faf82f183712226d0f28c"} Aug 04 16:02:25.574317 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:25.574298 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:02:25.574695 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:25.574681 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/0.log" Aug 04 16:02:25.574747 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:25.574708 2580 generic.go:356] "Generic (PLEG): container finished" podID="01d084ac-ce47-4d96-bbf0-2e78b83beb17" containerID="a099a48447971ea72d0e800a0bd282d352d69605b99a3cb76dbbf8e26c8c27fb" exitCode=255 Aug 04 16:02:25.574784 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:25.574772 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" event={"ID":"01d084ac-ce47-4d96-bbf0-2e78b83beb17","Type":"ContainerDied","Data":"a099a48447971ea72d0e800a0bd282d352d69605b99a3cb76dbbf8e26c8c27fb"} Aug 04 16:02:25.574824 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:25.574808 2580 scope.go:117] "RemoveContainer" containerID="b735854da0e268cae5f060460a73696b62cab6a6fae72bc95b51365fc716202d" Aug 04 16:02:25.575048 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:25.575035 2580 scope.go:117] "RemoveContainer" containerID="a099a48447971ea72d0e800a0bd282d352d69605b99a3cb76dbbf8e26c8c27fb" Aug 04 16:02:25.575238 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:25.575219 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-jcbjw_openshift-console-operator(01d084ac-ce47-4d96-bbf0-2e78b83beb17)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" podUID="01d084ac-ce47-4d96-bbf0-2e78b83beb17" Aug 04 16:02:25.586819 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:25.586781 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d8g5c" podStartSLOduration=4.095320278 podStartE2EDuration="7.586766064s" totalImagesPullingTime="3.491445786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:25.585857219 +0000 UTC m=+46.730189485" watchObservedRunningTime="2026-08-04 16:02:25.586766064 +0000 UTC m=+46.731098307" Aug 04 16:02:25.605602 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:25.605581 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2845_add717dd-d023-43b7-bf8b-f50516f053db/dns-node-resolver/0.log" Aug 04 16:02:26.582416 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:26.582377 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sdq7" event={"ID":"a9c0f306-ee82-4a4a-9680-5b0835964025","Type":"ContainerStarted","Data":"77095080ee5b701446557d8c001d97b5629a04ec55b4049889080d924b3b6878"} Aug 04 16:02:26.584017 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:26.583997 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:02:26.606547 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:26.606504 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8sdq7" podStartSLOduration=8.731230698 podStartE2EDuration="47.60648492s" totalImagesPullingTime="38.875254222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 16:01:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:26.60582453 +0000 UTC m=+47.750156778" watchObservedRunningTime="2026-08-04 16:02:26.60648492 +0000 UTC m=+47.750817166" Aug 04 16:02:27.001685 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:27.001623 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vbhgz_56b8e09d-90bb-401d-96a6-20d691168fc3/node-ca/0.log" Aug 04 16:02:27.401717 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:27.401689 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v8jrh_a9b9081e-ce41-4a27-97c9-d02d7bd2bab7/serve-healthcheck-canary/0.log" Aug 04 16:02:27.602825 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:27.602783 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-d8g5c_023c82e6-1623-4606-8fad-5eb6984f70f5/migrator/0.log" Aug 04 16:02:27.801952 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:27.801888 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-d8g5c_023c82e6-1623-4606-8fad-5eb6984f70f5/graceful-termination/0.log" Aug 04 16:02:28.004017 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.003941 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-v8nwn_cb89b27f-9d1b-41b0-830e-1e464d29d6f1/kube-storage-version-migrator-operator/0.log" Aug 04 16:02:28.103801 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.103773 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:28.106247 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.106222 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls\") pod \"image-registry-7c7ddb6d8b-g9s7p\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:28.168400 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.168375 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:28.204882 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.204864 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:28.204936 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.204894 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-rz5dr\" (UID: \"84092957-5deb-4cae-b1ba-310f24137f8e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:28.204980 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.204959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-n4zpv\" (UID: \"e9647849-9d1f-4b7e-8d2b-4697dadc4dbc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" Aug 04 16:02:28.205094 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:28.205077 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 16:02:28.205474 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:28.205150 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls podName:84092957-5deb-4cae-b1ba-310f24137f8e nodeName:}" failed. No retries permitted until 2026-08-04 16:02:44.205132304 +0000 UTC m=+65.349464550 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-rz5dr" (UID: "84092957-5deb-4cae-b1ba-310f24137f8e") : secret "cluster-monitoring-operator-tls" not found Aug 04 16:02:28.207178 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.207154 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9647849-9d1f-4b7e-8d2b-4697dadc4dbc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-n4zpv\" (UID: \"e9647849-9d1f-4b7e-8d2b-4697dadc4dbc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" Aug 04 16:02:28.207358 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.207337 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b557e1a0-4af6-4c29-bb50-214fa9d6882e-metrics-tls\") pod \"dns-default-zj6tj\" (UID: \"b557e1a0-4af6-4c29-bb50-214fa9d6882e\") " pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:28.264591 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.264561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:28.306068 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.306037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tcsrs\" (UID: \"6f235ea2-00aa-471c-b792-a2ed764869fa\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" Aug 04 16:02:28.306182 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.306080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:28.306182 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.306138 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:28.306319 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:28.306196 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 16:02:28.306319 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:28.306205 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle podName:66928c11-68ad-4888-8ee8-82a21af92095 nodeName:}" failed. No retries permitted until 2026-08-04 16:02:44.306185999 +0000 UTC m=+65.450518235 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle") pod "router-default-6d8f46644-nmgqm" (UID: "66928c11-68ad-4888-8ee8-82a21af92095") : configmap references non-existent config key: service-ca.crt Aug 04 16:02:28.306420 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:28.306277 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert podName:6f235ea2-00aa-471c-b792-a2ed764869fa nodeName:}" failed. No retries permitted until 2026-08-04 16:02:44.306258499 +0000 UTC m=+65.450590727 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-tcsrs" (UID: "6f235ea2-00aa-471c-b792-a2ed764869fa") : secret "networking-console-plugin-cert" not found Aug 04 16:02:28.308451 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.308428 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66928c11-68ad-4888-8ee8-82a21af92095-metrics-certs\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:28.334634 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.334574 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p"] Aug 04 16:02:28.340564 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:02:28.340533 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e539db_01a3_48e9_bf35_35807cebc38f.slice/crio-78401696b1b5fc29de7fda001fe73e78199c1140d1c1b07bfb045f635e7dfb9c WatchSource:0}: Error finding container 78401696b1b5fc29de7fda001fe73e78199c1140d1c1b07bfb045f635e7dfb9c: Status 404 returned error can't find the container with id 78401696b1b5fc29de7fda001fe73e78199c1140d1c1b07bfb045f635e7dfb9c Aug 04 16:02:28.422431 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.422400 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zj6tj"] Aug 04 16:02:28.424095 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:02:28.424069 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb557e1a0_4af6_4c29_bb50_214fa9d6882e.slice/crio-26f12bd8a4f24dafebc42de813dc839d8f1027bedead4e96bd5e57145733bccc WatchSource:0}: Error finding container 26f12bd8a4f24dafebc42de813dc839d8f1027bedead4e96bd5e57145733bccc: Status 404 returned error can't find the container with id 26f12bd8a4f24dafebc42de813dc839d8f1027bedead4e96bd5e57145733bccc Aug 04 16:02:28.496127 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.496106 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" Aug 04 16:02:28.592193 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.592159 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zj6tj" event={"ID":"b557e1a0-4af6-4c29-bb50-214fa9d6882e","Type":"ContainerStarted","Data":"26f12bd8a4f24dafebc42de813dc839d8f1027bedead4e96bd5e57145733bccc"} Aug 04 16:02:28.593895 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.593863 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" event={"ID":"64e539db-01a3-48e9-bf35-35807cebc38f","Type":"ContainerStarted","Data":"f27a55a7ad6a82964a52223efe7a56a04f42372b5db4c1637e4de4c79e4cbc7a"} Aug 04 16:02:28.594007 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.593915 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" event={"ID":"64e539db-01a3-48e9-bf35-35807cebc38f","Type":"ContainerStarted","Data":"78401696b1b5fc29de7fda001fe73e78199c1140d1c1b07bfb045f635e7dfb9c"} Aug 04 16:02:28.594007 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.593958 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:02:28.650405 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.650097 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" podStartSLOduration=48.650075044 podStartE2EDuration="48.650075044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:28.620470485 +0000 UTC m=+49.764802749" watchObservedRunningTime="2026-08-04 16:02:28.650075044 +0000 UTC m=+49.794407290" Aug 04 16:02:28.650405 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:28.650396 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv"] Aug 04 16:02:29.599008 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:29.598972 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" event={"ID":"e9647849-9d1f-4b7e-8d2b-4697dadc4dbc","Type":"ContainerStarted","Data":"d5c227c308feed4f4a5e0a30aaffb832ef43f4dae198376df59eadaa016aebf9"} Aug 04 16:02:30.604108 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:30.604066 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zj6tj" event={"ID":"b557e1a0-4af6-4c29-bb50-214fa9d6882e","Type":"ContainerStarted","Data":"adbaaf9a788d15ae9ccd9821f824c9904958ae1f54cc3383494e13a8a063f4af"} Aug 04 16:02:30.604578 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:30.604163 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zj6tj" event={"ID":"b557e1a0-4af6-4c29-bb50-214fa9d6882e","Type":"ContainerStarted","Data":"8ea179e75837f5ea21c10ed75a8cd9c0a8d8d15ea9250bd1ab87ace90b3977f1"} Aug 04 16:02:30.604578 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:30.604195 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:30.622714 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:30.622669 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zj6tj" podStartSLOduration=16.976871259 podStartE2EDuration="18.622654195s" totalImagesPullingTime="1.645782936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:30.621896564 +0000 UTC m=+51.766228833" watchObservedRunningTime="2026-08-04 16:02:30.622654195 +0000 UTC m=+51.766986441" Aug 04 16:02:31.609098 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:31.609070 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" event={"ID":"e9647849-9d1f-4b7e-8d2b-4697dadc4dbc","Type":"ContainerStarted","Data":"9a20a91ad14475f98bfbd1fea98505a4c43c3195991c41b41a23003dc2293e14"} Aug 04 16:02:32.614042 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:32.614008 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" event={"ID":"e9647849-9d1f-4b7e-8d2b-4697dadc4dbc","Type":"ContainerStarted","Data":"32a93afb562f8216caaa156386f30546412d63e0560ce62b816972f7272d6661"} Aug 04 16:02:32.615305 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:32.615276 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:32.615548 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:32.615536 2580 scope.go:117] "RemoveContainer" containerID="a099a48447971ea72d0e800a0bd282d352d69605b99a3cb76dbbf8e26c8c27fb" Aug 04 16:02:32.615686 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:32.615670 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-jcbjw_openshift-console-operator(01d084ac-ce47-4d96-bbf0-2e78b83beb17)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" podUID="01d084ac-ce47-4d96-bbf0-2e78b83beb17" Aug 04 16:02:32.632760 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:32.632718 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-n4zpv" podStartSLOduration=34.801115916 podStartE2EDuration="37.632706822s" totalImagesPullingTime="2.831590906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:32.632257434 +0000 UTC m=+53.776589707" watchObservedRunningTime="2026-08-04 16:02:32.632706822 +0000 UTC m=+53.777039066" Aug 04 16:02:33.557138 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:33.557102 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:33.617484 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:33.617457 2580 scope.go:117] "RemoveContainer" containerID="a099a48447971ea72d0e800a0bd282d352d69605b99a3cb76dbbf8e26c8c27fb" Aug 04 16:02:33.617821 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:02:33.617592 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-jcbjw_openshift-console-operator(01d084ac-ce47-4d96-bbf0-2e78b83beb17)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" podUID="01d084ac-ce47-4d96-bbf0-2e78b83beb17" Aug 04 16:02:37.469357 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:37.469326 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gjgb9" Aug 04 16:02:40.611010 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:40.610985 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zj6tj" Aug 04 16:02:44.031717 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.031664 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs\") pod \"network-metrics-daemon-t79jc\" (UID: \"56a1a012-b25b-4c19-b6be-73f382416c22\") " pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:02:44.034732 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.034715 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 16:02:44.044032 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.044009 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a1a012-b25b-4c19-b6be-73f382416c22-metrics-certs\") pod \"network-metrics-daemon-t79jc\" (UID: \"56a1a012-b25b-4c19-b6be-73f382416c22\") " pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:02:44.132670 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.132643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret\") pod \"global-pull-secret-syncer-5hhk7\" (UID: \"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db\") " pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:02:44.135515 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.135499 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 16:02:44.145705 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.145678 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9271e1-f3f4-4afc-b1ce-22d79e60a3db-original-pull-secret\") pod \"global-pull-secret-syncer-5hhk7\" (UID: \"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db\") " pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:02:44.156851 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.156835 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fv4gp\"" Aug 04 16:02:44.164108 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.164094 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79jc" Aug 04 16:02:44.234129 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.234096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhj2\" (UniqueName: \"kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2\") pod \"network-check-target-qvmnx\" (UID: \"fc84cbe7-6fbf-4152-9bcb-6227c4126473\") " pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:02:44.234249 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.234155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-rz5dr\" (UID: \"84092957-5deb-4cae-b1ba-310f24137f8e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:44.237200 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.237152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/84092957-5deb-4cae-b1ba-310f24137f8e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-rz5dr\" (UID: \"84092957-5deb-4cae-b1ba-310f24137f8e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:44.237338 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.237237 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxhj2\" (UniqueName: \"kubernetes.io/projected/fc84cbe7-6fbf-4152-9bcb-6227c4126473-kube-api-access-qxhj2\") pod \"network-check-target-qvmnx\" (UID: \"fc84cbe7-6fbf-4152-9bcb-6227c4126473\") " pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:02:44.281457 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.281433 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t79jc"] Aug 04 16:02:44.283347 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:02:44.283304 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a1a012_b25b_4c19_b6be_73f382416c22.slice/crio-0f8cfe9abde0a49e296642f022a2b88636630b7ce2f885b0ff153dd686613821 WatchSource:0}: Error finding container 0f8cfe9abde0a49e296642f022a2b88636630b7ce2f885b0ff153dd686613821: Status 404 returned error can't find the container with id 0f8cfe9abde0a49e296642f022a2b88636630b7ce2f885b0ff153dd686613821 Aug 04 16:02:44.335108 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.335086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tcsrs\" (UID: \"6f235ea2-00aa-471c-b792-a2ed764869fa\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" Aug 04 16:02:44.335210 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.335121 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:44.335712 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.335692 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66928c11-68ad-4888-8ee8-82a21af92095-service-ca-bundle\") pod \"router-default-6d8f46644-nmgqm\" (UID: \"66928c11-68ad-4888-8ee8-82a21af92095\") " pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:44.337191 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.337170 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f235ea2-00aa-471c-b792-a2ed764869fa-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tcsrs\" (UID: \"6f235ea2-00aa-471c-b792-a2ed764869fa\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" Aug 04 16:02:44.385585 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.385566 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cr8tk\"" Aug 04 16:02:44.393743 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.393728 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" Aug 04 16:02:44.442326 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.442298 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5hhk7" Aug 04 16:02:44.451610 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.451587 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kb5nq\"" Aug 04 16:02:44.459175 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.459151 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:02:44.477389 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.477246 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xpvpq\"" Aug 04 16:02:44.484982 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.484767 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" Aug 04 16:02:44.497430 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.496695 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6jz69\"" Aug 04 16:02:44.504200 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.504140 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:44.528305 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.524515 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr"] Aug 04 16:02:44.529482 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:02:44.529454 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84092957_5deb_4cae_b1ba_310f24137f8e.slice/crio-6ea91018a5321f485d0c3bd7a81ab1fc4bbaaf404d568cb373ca2f0c791afce1 WatchSource:0}: Error finding container 6ea91018a5321f485d0c3bd7a81ab1fc4bbaaf404d568cb373ca2f0c791afce1: Status 404 returned error can't find the container with id 6ea91018a5321f485d0c3bd7a81ab1fc4bbaaf404d568cb373ca2f0c791afce1 Aug 04 16:02:44.594806 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.594777 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5hhk7"] Aug 04 16:02:44.600988 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:02:44.600934 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9271e1_f3f4_4afc_b1ce_22d79e60a3db.slice/crio-baede3a1d67c203e97da2a5b10d6cc89db73e0cac2f728fb7d0b698a94dacfc7 WatchSource:0}: Error finding container baede3a1d67c203e97da2a5b10d6cc89db73e0cac2f728fb7d0b698a94dacfc7: Status 404 returned error can't find the container with id baede3a1d67c203e97da2a5b10d6cc89db73e0cac2f728fb7d0b698a94dacfc7 Aug 04 16:02:44.627822 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.627796 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qvmnx"] Aug 04 16:02:44.630562 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:02:44.630526 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc84cbe7_6fbf_4152_9bcb_6227c4126473.slice/crio-ec1e0a616f9fa3b877f57fb0534d60a9230b871daa5ee8b4e87f914f3eac68ea WatchSource:0}: Error finding container ec1e0a616f9fa3b877f57fb0534d60a9230b871daa5ee8b4e87f914f3eac68ea: Status 404 returned error can't find the container with id ec1e0a616f9fa3b877f57fb0534d60a9230b871daa5ee8b4e87f914f3eac68ea Aug 04 16:02:44.641443 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.641418 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs"] Aug 04 16:02:44.643875 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:02:44.643851 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f235ea2_00aa_471c_b792_a2ed764869fa.slice/crio-4b8494c9c3437d5d168c2dee3dc65114802cff607acbce26d41128b1de67dfd1 WatchSource:0}: Error finding container 4b8494c9c3437d5d168c2dee3dc65114802cff607acbce26d41128b1de67dfd1: Status 404 returned error can't find the container with id 4b8494c9c3437d5d168c2dee3dc65114802cff607acbce26d41128b1de67dfd1 Aug 04 16:02:44.652615 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.652584 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5hhk7" event={"ID":"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db","Type":"ContainerStarted","Data":"baede3a1d67c203e97da2a5b10d6cc89db73e0cac2f728fb7d0b698a94dacfc7"} Aug 04 16:02:44.653676 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.653644 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" event={"ID":"84092957-5deb-4cae-b1ba-310f24137f8e","Type":"ContainerStarted","Data":"6ea91018a5321f485d0c3bd7a81ab1fc4bbaaf404d568cb373ca2f0c791afce1"} Aug 04 16:02:44.654808 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.654787 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" event={"ID":"6f235ea2-00aa-471c-b792-a2ed764869fa","Type":"ContainerStarted","Data":"4b8494c9c3437d5d168c2dee3dc65114802cff607acbce26d41128b1de67dfd1"} Aug 04 16:02:44.655864 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.655844 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qvmnx" event={"ID":"fc84cbe7-6fbf-4152-9bcb-6227c4126473","Type":"ContainerStarted","Data":"ec1e0a616f9fa3b877f57fb0534d60a9230b871daa5ee8b4e87f914f3eac68ea"} Aug 04 16:02:44.656991 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.656973 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t79jc" event={"ID":"56a1a012-b25b-4c19-b6be-73f382416c22","Type":"ContainerStarted","Data":"0f8cfe9abde0a49e296642f022a2b88636630b7ce2f885b0ff153dd686613821"} Aug 04 16:02:44.685572 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:44.685548 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d8f46644-nmgqm"] Aug 04 16:02:44.715098 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:02:44.715068 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66928c11_68ad_4888_8ee8_82a21af92095.slice/crio-7f18417f02b841c0c1649baff9dda7cedf9d63510c9c9a9d89ec870f83ea8bb6 WatchSource:0}: Error finding container 7f18417f02b841c0c1649baff9dda7cedf9d63510c9c9a9d89ec870f83ea8bb6: Status 404 returned error can't find the container with id 7f18417f02b841c0c1649baff9dda7cedf9d63510c9c9a9d89ec870f83ea8bb6 Aug 04 16:02:45.663330 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:45.663276 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d8f46644-nmgqm" event={"ID":"66928c11-68ad-4888-8ee8-82a21af92095","Type":"ContainerStarted","Data":"3a2a363d9dab01c0497c01c9066c805370f618725686480f5f18c4bdc018a1da"} Aug 04 16:02:45.663858 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:45.663355 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d8f46644-nmgqm" event={"ID":"66928c11-68ad-4888-8ee8-82a21af92095","Type":"ContainerStarted","Data":"7f18417f02b841c0c1649baff9dda7cedf9d63510c9c9a9d89ec870f83ea8bb6"} Aug 04 16:02:45.665665 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:45.665623 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qvmnx" event={"ID":"fc84cbe7-6fbf-4152-9bcb-6227c4126473","Type":"ContainerStarted","Data":"4988ccff2dbf4dbede2fb081f89aa5455446fde992e5ed9b2b650122819688c7"} Aug 04 16:02:45.665981 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:45.665957 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:02:45.689543 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:45.689493 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6d8f46644-nmgqm" podStartSLOduration=50.689475683 podStartE2EDuration="50.689475683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:45.687535053 +0000 UTC m=+66.831867331" watchObservedRunningTime="2026-08-04 16:02:45.689475683 +0000 UTC m=+66.833807929" Aug 04 16:02:46.505082 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:46.505047 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:46.508172 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:46.508143 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:46.532468 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:46.532415 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qvmnx" podStartSLOduration=67.53239578 podStartE2EDuration="1m7.53239578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:45.704653223 +0000 UTC m=+66.848985490" watchObservedRunningTime="2026-08-04 16:02:46.53239578 +0000 UTC m=+67.676728027" Aug 04 16:02:46.670113 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:46.670076 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:46.671497 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:46.671475 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6d8f46644-nmgqm" Aug 04 16:02:47.313549 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:47.313518 2580 scope.go:117] "RemoveContainer" containerID="a099a48447971ea72d0e800a0bd282d352d69605b99a3cb76dbbf8e26c8c27fb" Aug 04 16:02:48.173025 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:48.172988 2580 patch_prober.go:28] interesting pod/image-registry-7c7ddb6d8b-g9s7p container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 16:02:48.173420 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:48.173064 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:02:49.603756 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:49.603721 2580 patch_prober.go:28] interesting pod/image-registry-7c7ddb6d8b-g9s7p container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 16:02:49.604093 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:49.603773 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:02:50.686409 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:50.686371 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" event={"ID":"6f235ea2-00aa-471c-b792-a2ed764869fa","Type":"ContainerStarted","Data":"b715eadf5bbb59b0146fdef1d5a78207572e87690ddfed405984907dce992a45"} Aug 04 16:02:50.688263 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:50.688244 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:02:50.688391 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:50.688344 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" event={"ID":"01d084ac-ce47-4d96-bbf0-2e78b83beb17","Type":"ContainerStarted","Data":"1daf631ded35365b2e4508f086a4e7247baa464bc9dff33a609a61bf4adfd929"} Aug 04 16:02:50.688624 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:50.688603 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:50.689918 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:50.689888 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5hhk7" event={"ID":"cb9271e1-f3f4-4afc-b1ce-22d79e60a3db","Type":"ContainerStarted","Data":"8602f5aa7f7ac8258ba0615a1c039407d0ec0fa2cc1b5fccc562c01e6ffd0e7d"} Aug 04 16:02:50.691950 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:50.691922 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t79jc" event={"ID":"56a1a012-b25b-4c19-b6be-73f382416c22","Type":"ContainerStarted","Data":"d4c1b2d72e036c0cbfd668129c8ad5045aee728e6b43e1511f0fd9dde4364357"} Aug 04 16:02:50.692047 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:50.691967 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t79jc" event={"ID":"56a1a012-b25b-4c19-b6be-73f382416c22","Type":"ContainerStarted","Data":"14235b5ed7106976452da7faa91b5c4d23dd59218b0cd3d92e8262250ce11e54"} Aug 04 16:02:50.693624 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:50.693602 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" event={"ID":"84092957-5deb-4cae-b1ba-310f24137f8e","Type":"ContainerStarted","Data":"fd98fee774bfb1e201bfff8a41a5b8ac8e6785fcba7bc5fedd0b3812babac5f4"} Aug 04 16:02:50.693783 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:50.693767 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-jcbjw" Aug 04 16:02:50.706158 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:50.706111 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-tcsrs" podStartSLOduration=37.512840693 podStartE2EDuration="42.706099104s" totalImagesPullingTime="5.193258411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:50.704919825 +0000 UTC m=+71.849252103" watchObservedRunningTime="2026-08-04 16:02:50.706099104 +0000 UTC m=+71.850431374" Aug 04 16:02:50.721205 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:50.721169 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-t79jc" podStartSLOduration=66.167400043 podStartE2EDuration="1m11.721153158s" totalImagesPullingTime="5.553753115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:50.720848074 +0000 UTC m=+71.865180339" watchObservedRunningTime="2026-08-04 16:02:50.721153158 +0000 UTC m=+71.865485405" Aug 04 16:02:50.743569 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:50.743533 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-rz5dr" podStartSLOduration=50.440314927 podStartE2EDuration="55.743522468s" totalImagesPullingTime="5.303207541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:50.741896999 +0000 UTC m=+71.886229253" watchObservedRunningTime="2026-08-04 16:02:50.743522468 +0000 UTC m=+71.887854713" Aug 04 16:02:50.759785 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:50.759749 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5hhk7" podStartSLOduration=66.513450148 podStartE2EDuration="1m11.759738713s" totalImagesPullingTime="5.246288565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 16:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:50.758799697 +0000 UTC m=+71.903131942" watchObservedRunningTime="2026-08-04 16:02:50.759738713 +0000 UTC m=+71.904070987" Aug 04 16:02:58.173739 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:58.173705 2580 patch_prober.go:28] interesting pod/image-registry-7c7ddb6d8b-g9s7p container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 16:02:58.174179 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:58.173763 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:02:59.602853 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:59.602816 2580 patch_prober.go:28] interesting pod/image-registry-7c7ddb6d8b-g9s7p container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 16:02:59.603180 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:02:59.602874 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:03:08.172386 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:08.172240 2580 patch_prober.go:28] interesting pod/image-registry-7c7ddb6d8b-g9s7p container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 16:03:08.172386 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:08.172329 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:03:08.172386 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:08.172378 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:03:08.172851 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:08.172816 2580 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"f27a55a7ad6a82964a52223efe7a56a04f42372b5db4c1637e4de4c79e4cbc7a"} pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" containerMessage="Container registry failed liveness probe, will be restarted" Aug 04 16:03:08.176171 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:08.176145 2580 patch_prober.go:28] interesting pod/image-registry-7c7ddb6d8b-g9s7p container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 16:03:08.176303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:08.176194 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:03:16.672635 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:16.672606 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qvmnx" Aug 04 16:03:18.176869 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:18.176838 2580 patch_prober.go:28] interesting pod/image-registry-7c7ddb6d8b-g9s7p container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 16:03:18.177238 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:18.176911 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:03:21.802293 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:21.802211 2580 generic.go:356] "Generic (PLEG): container finished" podID="687ce5a4-b2e9-4c41-bb89-89d423b1e514" containerID="9cb23de8d0ef8a37fbdb130eb28c4789cfe9c844dc7f6cadfe80c132f258310d" exitCode=0 Aug 04 16:03:21.802600 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:21.802304 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp" event={"ID":"687ce5a4-b2e9-4c41-bb89-89d423b1e514","Type":"ContainerDied","Data":"9cb23de8d0ef8a37fbdb130eb28c4789cfe9c844dc7f6cadfe80c132f258310d"} Aug 04 16:03:21.802643 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:21.802626 2580 scope.go:117] "RemoveContainer" containerID="9cb23de8d0ef8a37fbdb130eb28c4789cfe9c844dc7f6cadfe80c132f258310d" Aug 04 16:03:22.806997 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:22.806965 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sg8sp" event={"ID":"687ce5a4-b2e9-4c41-bb89-89d423b1e514","Type":"ContainerStarted","Data":"0624b2e76c33970daeaf427253d911a67f078f3cba7b6d13dcb2713e3b1f8fee"} Aug 04 16:03:28.177091 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:28.177060 2580 patch_prober.go:28] interesting pod/image-registry-7c7ddb6d8b-g9s7p container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 16:03:28.177547 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:28.177108 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:03:33.190687 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:33.190643 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" containerName="registry" containerID="cri-o://f27a55a7ad6a82964a52223efe7a56a04f42372b5db4c1637e4de4c79e4cbc7a" gracePeriod=30 Aug 04 16:03:33.853451 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:33.853418 2580 generic.go:356] "Generic (PLEG): container finished" podID="64e539db-01a3-48e9-bf35-35807cebc38f" containerID="f27a55a7ad6a82964a52223efe7a56a04f42372b5db4c1637e4de4c79e4cbc7a" exitCode=0 Aug 04 16:03:33.853614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:33.853496 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" event={"ID":"64e539db-01a3-48e9-bf35-35807cebc38f","Type":"ContainerDied","Data":"f27a55a7ad6a82964a52223efe7a56a04f42372b5db4c1637e4de4c79e4cbc7a"} Aug 04 16:03:33.853614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:33.853569 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" event={"ID":"64e539db-01a3-48e9-bf35-35807cebc38f","Type":"ContainerStarted","Data":"d7fdab947f24fd826b8b083bfb936a2e57a7e51ecebfa010c84b5cb995433db1"} Aug 04 16:03:33.853832 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:33.853815 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:03:36.865506 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:36.865474 2580 generic.go:356] "Generic (PLEG): container finished" podID="cb89b27f-9d1b-41b0-830e-1e464d29d6f1" containerID="b178d7a652b408c67d8346a9d89b38e8888f945a93cdf5abf8175d01b858f113" exitCode=0 Aug 04 16:03:36.865840 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:36.865537 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn" event={"ID":"cb89b27f-9d1b-41b0-830e-1e464d29d6f1","Type":"ContainerDied","Data":"b178d7a652b408c67d8346a9d89b38e8888f945a93cdf5abf8175d01b858f113"} Aug 04 16:03:36.865840 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:36.865820 2580 scope.go:117] "RemoveContainer" containerID="b178d7a652b408c67d8346a9d89b38e8888f945a93cdf5abf8175d01b858f113" Aug 04 16:03:37.871559 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:37.871528 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v8nwn" event={"ID":"cb89b27f-9d1b-41b0-830e-1e464d29d6f1","Type":"ContainerStarted","Data":"663cd3f9841c9d095815609144842f7d50a832cae0b011937a6611445c7e86b2"} Aug 04 16:03:41.731927 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.731883 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg"] Aug 04 16:03:41.734873 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.734852 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg" Aug 04 16:03:41.740662 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.740637 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Aug 04 16:03:41.740801 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.740668 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Aug 04 16:03:41.741061 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.741038 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Aug 04 16:03:41.741182 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.741123 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Aug 04 16:03:41.757237 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.757216 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q"] Aug 04 16:03:41.759685 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.759665 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.763914 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.763849 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Aug 04 16:03:41.765145 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.765123 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg"] Aug 04 16:03:41.767664 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.767645 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Aug 04 16:03:41.771611 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.771592 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Aug 04 16:03:41.771774 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.771601 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Aug 04 16:03:41.788512 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.788493 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf"] Aug 04 16:03:41.795072 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.795049 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf" Aug 04 16:03:41.798583 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.798565 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-nhpx2\"" Aug 04 16:03:41.799126 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.799109 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Aug 04 16:03:41.805337 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.805316 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q"] Aug 04 16:03:41.815802 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.815780 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf"] Aug 04 16:03:41.824870 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.824847 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc8tx\" (UniqueName: \"kubernetes.io/projected/97c4bc8f-306d-41b3-bd01-16fbb9e475a7-kube-api-access-rc8tx\") pod \"managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf\" (UID: \"97c4bc8f-306d-41b3-bd01-16fbb9e475a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf" Aug 04 16:03:41.824974 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.824885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97c4bc8f-306d-41b3-bd01-16fbb9e475a7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf\" (UID: \"97c4bc8f-306d-41b3-bd01-16fbb9e475a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf" Aug 04 16:03:41.824974 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.824923 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a41c1a1a-6e8a-44d6-ae90-47b0763493be-hub\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.825089 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.824993 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcd2j\" (UniqueName: \"kubernetes.io/projected/a41c1a1a-6e8a-44d6-ae90-47b0763493be-kube-api-access-dcd2j\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.825089 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.825023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a41c1a1a-6e8a-44d6-ae90-47b0763493be-ca\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.825089 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.825040 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a41c1a1a-6e8a-44d6-ae90-47b0763493be-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.825089 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.825059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc6d6\" (UniqueName: \"kubernetes.io/projected/27fd7d60-16e2-4dcb-838c-ace287a26c21-kube-api-access-bc6d6\") pod \"klusterlet-addon-workmgr-686789cd78-dp8fg\" (UID: \"27fd7d60-16e2-4dcb-838c-ace287a26c21\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg" Aug 04 16:03:41.825089 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.825082 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/27fd7d60-16e2-4dcb-838c-ace287a26c21-klusterlet-config\") pod \"klusterlet-addon-workmgr-686789cd78-dp8fg\" (UID: \"27fd7d60-16e2-4dcb-838c-ace287a26c21\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg" Aug 04 16:03:41.825853 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.825132 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a41c1a1a-6e8a-44d6-ae90-47b0763493be-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.825853 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.825168 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41c1a1a-6e8a-44d6-ae90-47b0763493be-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.825853 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.825222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27fd7d60-16e2-4dcb-838c-ace287a26c21-tmp\") pod \"klusterlet-addon-workmgr-686789cd78-dp8fg\" (UID: \"27fd7d60-16e2-4dcb-838c-ace287a26c21\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg" Aug 04 16:03:41.858908 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.858128 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dvdq8"] Aug 04 16:03:41.860774 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.860559 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dvdq8" Aug 04 16:03:41.863576 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.863549 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Aug 04 16:03:41.863687 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.863574 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2t8f9\"" Aug 04 16:03:41.870586 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.870563 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dvdq8"] Aug 04 16:03:41.927183 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.926441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/27fd7d60-16e2-4dcb-838c-ace287a26c21-klusterlet-config\") pod \"klusterlet-addon-workmgr-686789cd78-dp8fg\" (UID: \"27fd7d60-16e2-4dcb-838c-ace287a26c21\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg" Aug 04 16:03:41.927183 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.926488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a41c1a1a-6e8a-44d6-ae90-47b0763493be-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.927183 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.926515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41c1a1a-6e8a-44d6-ae90-47b0763493be-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.927183 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.926548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27fd7d60-16e2-4dcb-838c-ace287a26c21-tmp\") pod \"klusterlet-addon-workmgr-686789cd78-dp8fg\" (UID: \"27fd7d60-16e2-4dcb-838c-ace287a26c21\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg" Aug 04 16:03:41.927183 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.926609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/031a5dd3-4e36-4920-b9a3-d732b8a0a0f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-dvdq8\" (UID: \"031a5dd3-4e36-4920-b9a3-d732b8a0a0f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dvdq8" Aug 04 16:03:41.927183 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.926642 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc8tx\" (UniqueName: \"kubernetes.io/projected/97c4bc8f-306d-41b3-bd01-16fbb9e475a7-kube-api-access-rc8tx\") pod \"managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf\" (UID: \"97c4bc8f-306d-41b3-bd01-16fbb9e475a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf" Aug 04 16:03:41.927183 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.926670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97c4bc8f-306d-41b3-bd01-16fbb9e475a7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf\" (UID: \"97c4bc8f-306d-41b3-bd01-16fbb9e475a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf" Aug 04 16:03:41.927183 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.926712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a41c1a1a-6e8a-44d6-ae90-47b0763493be-hub\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.927183 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.926754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcd2j\" (UniqueName: \"kubernetes.io/projected/a41c1a1a-6e8a-44d6-ae90-47b0763493be-kube-api-access-dcd2j\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.927183 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.926780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a41c1a1a-6e8a-44d6-ae90-47b0763493be-ca\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.927183 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.926808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a41c1a1a-6e8a-44d6-ae90-47b0763493be-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.927183 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.926844 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc6d6\" (UniqueName: \"kubernetes.io/projected/27fd7d60-16e2-4dcb-838c-ace287a26c21-kube-api-access-bc6d6\") pod \"klusterlet-addon-workmgr-686789cd78-dp8fg\" (UID: \"27fd7d60-16e2-4dcb-838c-ace287a26c21\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg" Aug 04 16:03:41.929141 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.928951 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27fd7d60-16e2-4dcb-838c-ace287a26c21-tmp\") pod \"klusterlet-addon-workmgr-686789cd78-dp8fg\" (UID: \"27fd7d60-16e2-4dcb-838c-ace287a26c21\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg" Aug 04 16:03:41.929141 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.929088 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41c1a1a-6e8a-44d6-ae90-47b0763493be-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.929973 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.929558 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/27fd7d60-16e2-4dcb-838c-ace287a26c21-klusterlet-config\") pod \"klusterlet-addon-workmgr-686789cd78-dp8fg\" (UID: \"27fd7d60-16e2-4dcb-838c-ace287a26c21\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg" Aug 04 16:03:41.930576 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.930544 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a41c1a1a-6e8a-44d6-ae90-47b0763493be-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.930754 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.930736 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97c4bc8f-306d-41b3-bd01-16fbb9e475a7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf\" (UID: \"97c4bc8f-306d-41b3-bd01-16fbb9e475a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf" Aug 04 16:03:41.931475 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.931449 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a41c1a1a-6e8a-44d6-ae90-47b0763493be-ca\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.931592 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.931573 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a41c1a1a-6e8a-44d6-ae90-47b0763493be-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.931717 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.931699 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a41c1a1a-6e8a-44d6-ae90-47b0763493be-hub\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.938736 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.938704 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc6d6\" (UniqueName: \"kubernetes.io/projected/27fd7d60-16e2-4dcb-838c-ace287a26c21-kube-api-access-bc6d6\") pod \"klusterlet-addon-workmgr-686789cd78-dp8fg\" (UID: \"27fd7d60-16e2-4dcb-838c-ace287a26c21\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg" Aug 04 16:03:41.939797 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.939750 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcd2j\" (UniqueName: \"kubernetes.io/projected/a41c1a1a-6e8a-44d6-ae90-47b0763493be-kube-api-access-dcd2j\") pod \"cluster-proxy-proxy-agent-5597d5df7-bqc8q\" (UID: \"a41c1a1a-6e8a-44d6-ae90-47b0763493be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:41.940786 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.940757 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc8tx\" (UniqueName: \"kubernetes.io/projected/97c4bc8f-306d-41b3-bd01-16fbb9e475a7-kube-api-access-rc8tx\") pod \"managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf\" (UID: \"97c4bc8f-306d-41b3-bd01-16fbb9e475a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf" Aug 04 16:03:41.967093 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.967073 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-kw7q7"] Aug 04 16:03:41.970451 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.970430 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-577c6684f8-9sf6c"] Aug 04 16:03:41.970573 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.970556 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-kw7q7" Aug 04 16:03:41.972634 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.972615 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t7vhm"] Aug 04 16:03:41.972755 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.972739 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:41.974062 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.974042 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hmdv4\"" Aug 04 16:03:41.974062 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.974049 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 16:03:41.974175 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.974093 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 16:03:41.974857 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.974841 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:41.975218 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.975187 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 04 16:03:41.975483 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.975468 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 04 16:03:41.975538 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.975472 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 04 16:03:41.975708 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.975691 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bp8m6\"" Aug 04 16:03:41.975813 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.975732 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 04 16:03:41.975813 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.975730 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 04 16:03:41.977778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.977685 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-svpvj\"" Aug 04 16:03:41.977778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.977762 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 16:03:41.977969 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.977930 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 16:03:41.982702 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.982659 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 04 16:03:41.985117 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.985098 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-kw7q7"] Aug 04 16:03:41.987349 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.987331 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-577c6684f8-9sf6c"] Aug 04 16:03:41.989115 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:41.989096 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t7vhm"] Aug 04 16:03:42.027518 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.027494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4081f9ba-ac7b-4f25-b676-4af064e18d99-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7vhm\" (UID: \"4081f9ba-ac7b-4f25-b676-4af064e18d99\") " pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:42.027629 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.027522 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4081f9ba-ac7b-4f25-b676-4af064e18d99-crio-socket\") pod \"insights-runtime-extractor-t7vhm\" (UID: \"4081f9ba-ac7b-4f25-b676-4af064e18d99\") " pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:42.027629 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.027541 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-oauth-serving-cert\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.027629 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.027561 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp8zb\" (UniqueName: \"kubernetes.io/projected/41a9698f-7fff-4a1e-915a-bb1875387c14-kube-api-access-jp8zb\") pod \"downloads-77c974ffcf-kw7q7\" (UID: \"41a9698f-7fff-4a1e-915a-bb1875387c14\") " pod="openshift-console/downloads-77c974ffcf-kw7q7" Aug 04 16:03:42.027788 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.027636 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/031a5dd3-4e36-4920-b9a3-d732b8a0a0f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-dvdq8\" (UID: \"031a5dd3-4e36-4920-b9a3-d732b8a0a0f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dvdq8" Aug 04 16:03:42.027788 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.027681 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4081f9ba-ac7b-4f25-b676-4af064e18d99-data-volume\") pod \"insights-runtime-extractor-t7vhm\" (UID: \"4081f9ba-ac7b-4f25-b676-4af064e18d99\") " pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:42.027788 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.027711 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64jxx\" (UniqueName: \"kubernetes.io/projected/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-kube-api-access-64jxx\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.027788 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.027737 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-oauth-config\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.027953 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.027789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-config\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.027953 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.027832 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4081f9ba-ac7b-4f25-b676-4af064e18d99-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7vhm\" (UID: \"4081f9ba-ac7b-4f25-b676-4af064e18d99\") " pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:42.027953 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.027863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-trusted-ca-bundle\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.027953 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.027886 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-serving-cert\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.027953 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.027944 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-service-ca\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.028121 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.027977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhq9v\" (UniqueName: \"kubernetes.io/projected/4081f9ba-ac7b-4f25-b676-4af064e18d99-kube-api-access-zhq9v\") pod \"insights-runtime-extractor-t7vhm\" (UID: \"4081f9ba-ac7b-4f25-b676-4af064e18d99\") " pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:42.029924 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.029908 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/031a5dd3-4e36-4920-b9a3-d732b8a0a0f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-dvdq8\" (UID: \"031a5dd3-4e36-4920-b9a3-d732b8a0a0f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dvdq8" Aug 04 16:03:42.046702 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.046679 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg" Aug 04 16:03:42.079250 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.079214 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" Aug 04 16:03:42.104562 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.104514 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf" Aug 04 16:03:42.129121 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.129088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp8zb\" (UniqueName: \"kubernetes.io/projected/41a9698f-7fff-4a1e-915a-bb1875387c14-kube-api-access-jp8zb\") pod \"downloads-77c974ffcf-kw7q7\" (UID: \"41a9698f-7fff-4a1e-915a-bb1875387c14\") " pod="openshift-console/downloads-77c974ffcf-kw7q7" Aug 04 16:03:42.129242 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.129148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4081f9ba-ac7b-4f25-b676-4af064e18d99-data-volume\") pod \"insights-runtime-extractor-t7vhm\" (UID: \"4081f9ba-ac7b-4f25-b676-4af064e18d99\") " pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:42.129242 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.129184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64jxx\" (UniqueName: \"kubernetes.io/projected/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-kube-api-access-64jxx\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.129242 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.129215 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-oauth-config\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.129608 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.129252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-config\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.129608 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.129303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4081f9ba-ac7b-4f25-b676-4af064e18d99-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7vhm\" (UID: \"4081f9ba-ac7b-4f25-b676-4af064e18d99\") " pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:42.129608 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.129337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-trusted-ca-bundle\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.129608 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.129363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-serving-cert\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.129608 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.129410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-service-ca\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.129608 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.129443 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhq9v\" (UniqueName: \"kubernetes.io/projected/4081f9ba-ac7b-4f25-b676-4af064e18d99-kube-api-access-zhq9v\") pod \"insights-runtime-extractor-t7vhm\" (UID: \"4081f9ba-ac7b-4f25-b676-4af064e18d99\") " pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:42.129608 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.129494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4081f9ba-ac7b-4f25-b676-4af064e18d99-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7vhm\" (UID: \"4081f9ba-ac7b-4f25-b676-4af064e18d99\") " pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:42.129608 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.129521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4081f9ba-ac7b-4f25-b676-4af064e18d99-crio-socket\") pod \"insights-runtime-extractor-t7vhm\" (UID: \"4081f9ba-ac7b-4f25-b676-4af064e18d99\") " pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:42.129608 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.129551 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-oauth-serving-cert\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.130375 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.130340 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-oauth-serving-cert\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.132699 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.131076 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-trusted-ca-bundle\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.132699 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.131504 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4081f9ba-ac7b-4f25-b676-4af064e18d99-data-volume\") pod \"insights-runtime-extractor-t7vhm\" (UID: \"4081f9ba-ac7b-4f25-b676-4af064e18d99\") " pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:42.132699 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.132089 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-config\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.132699 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.132645 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4081f9ba-ac7b-4f25-b676-4af064e18d99-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7vhm\" (UID: \"4081f9ba-ac7b-4f25-b676-4af064e18d99\") " pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:42.132966 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.132734 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4081f9ba-ac7b-4f25-b676-4af064e18d99-crio-socket\") pod \"insights-runtime-extractor-t7vhm\" (UID: \"4081f9ba-ac7b-4f25-b676-4af064e18d99\") " pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:42.136646 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.136598 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-oauth-config\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.139122 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.138832 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-service-ca\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.141732 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.140842 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4081f9ba-ac7b-4f25-b676-4af064e18d99-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7vhm\" (UID: \"4081f9ba-ac7b-4f25-b676-4af064e18d99\") " pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:42.141732 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.141484 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp8zb\" (UniqueName: \"kubernetes.io/projected/41a9698f-7fff-4a1e-915a-bb1875387c14-kube-api-access-jp8zb\") pod \"downloads-77c974ffcf-kw7q7\" (UID: \"41a9698f-7fff-4a1e-915a-bb1875387c14\") " pod="openshift-console/downloads-77c974ffcf-kw7q7" Aug 04 16:03:42.142549 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.142521 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-serving-cert\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.143617 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.143561 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64jxx\" (UniqueName: \"kubernetes.io/projected/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-kube-api-access-64jxx\") pod \"console-577c6684f8-9sf6c\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.143956 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.143928 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhq9v\" (UniqueName: \"kubernetes.io/projected/4081f9ba-ac7b-4f25-b676-4af064e18d99-kube-api-access-zhq9v\") pod \"insights-runtime-extractor-t7vhm\" (UID: \"4081f9ba-ac7b-4f25-b676-4af064e18d99\") " pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:42.180748 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.180426 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dvdq8" Aug 04 16:03:42.198112 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.198085 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg"] Aug 04 16:03:42.200172 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:03:42.200132 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fd7d60_16e2_4dcb_838c_ace287a26c21.slice/crio-a901dfac46130eefa4e4b4028ac5334f5535ed6b825dd7ce481fbe008de69fe4 WatchSource:0}: Error finding container a901dfac46130eefa4e4b4028ac5334f5535ed6b825dd7ce481fbe008de69fe4: Status 404 returned error can't find the container with id a901dfac46130eefa4e4b4028ac5334f5535ed6b825dd7ce481fbe008de69fe4 Aug 04 16:03:42.227074 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.227036 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q"] Aug 04 16:03:42.264620 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.264592 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf"] Aug 04 16:03:42.267439 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:03:42.267406 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97c4bc8f_306d_41b3_bd01_16fbb9e475a7.slice/crio-f9b19e9bc6bb0e8e557be083d3c28b5110ee1d1e03804a764519f2f83aec7c11 WatchSource:0}: Error finding container f9b19e9bc6bb0e8e557be083d3c28b5110ee1d1e03804a764519f2f83aec7c11: Status 404 returned error can't find the container with id f9b19e9bc6bb0e8e557be083d3c28b5110ee1d1e03804a764519f2f83aec7c11 Aug 04 16:03:42.281309 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.281275 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-kw7q7" Aug 04 16:03:42.290995 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.290964 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:03:42.297955 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.297934 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t7vhm" Aug 04 16:03:42.320212 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.320084 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dvdq8"] Aug 04 16:03:42.323321 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:03:42.323291 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031a5dd3_4e36_4920_b9a3_d732b8a0a0f5.slice/crio-3d8bea835340cb85858038579faeb993411d8198375e0cfde56094e0520cbc71 WatchSource:0}: Error finding container 3d8bea835340cb85858038579faeb993411d8198375e0cfde56094e0520cbc71: Status 404 returned error can't find the container with id 3d8bea835340cb85858038579faeb993411d8198375e0cfde56094e0520cbc71 Aug 04 16:03:42.433331 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.431417 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-kw7q7"] Aug 04 16:03:42.439390 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:03:42.439360 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a9698f_7fff_4a1e_915a_bb1875387c14.slice/crio-7c492156ec81a6a02a0ae21316b17ad6ce6dee833f6a58f70be1f426f1b2d67d WatchSource:0}: Error finding container 7c492156ec81a6a02a0ae21316b17ad6ce6dee833f6a58f70be1f426f1b2d67d: Status 404 returned error can't find the container with id 7c492156ec81a6a02a0ae21316b17ad6ce6dee833f6a58f70be1f426f1b2d67d Aug 04 16:03:42.449949 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.449927 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-577c6684f8-9sf6c"] Aug 04 16:03:42.452470 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:03:42.452443 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684f9c20_4cdb_40c0_9350_4495bf1fe6c1.slice/crio-09598eaaf3050ddf9fb8fdac291c97f30f6471e89e6ea44e60db5eed0dc57d08 WatchSource:0}: Error finding container 09598eaaf3050ddf9fb8fdac291c97f30f6471e89e6ea44e60db5eed0dc57d08: Status 404 returned error can't find the container with id 09598eaaf3050ddf9fb8fdac291c97f30f6471e89e6ea44e60db5eed0dc57d08 Aug 04 16:03:42.469910 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.469891 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t7vhm"] Aug 04 16:03:42.471668 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:03:42.471649 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4081f9ba_ac7b_4f25_b676_4af064e18d99.slice/crio-162ef6b9ad91aad453e09f3ab84cca470717fef0e141d390fa4868d5345e05e9 WatchSource:0}: Error finding container 162ef6b9ad91aad453e09f3ab84cca470717fef0e141d390fa4868d5345e05e9: Status 404 returned error can't find the container with id 162ef6b9ad91aad453e09f3ab84cca470717fef0e141d390fa4868d5345e05e9 Aug 04 16:03:42.889547 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.889513 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-577c6684f8-9sf6c" event={"ID":"684f9c20-4cdb-40c0-9350-4495bf1fe6c1","Type":"ContainerStarted","Data":"09598eaaf3050ddf9fb8fdac291c97f30f6471e89e6ea44e60db5eed0dc57d08"} Aug 04 16:03:42.890641 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.890618 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dvdq8" event={"ID":"031a5dd3-4e36-4920-b9a3-d732b8a0a0f5","Type":"ContainerStarted","Data":"3d8bea835340cb85858038579faeb993411d8198375e0cfde56094e0520cbc71"} Aug 04 16:03:42.891536 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.891516 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-kw7q7" event={"ID":"41a9698f-7fff-4a1e-915a-bb1875387c14","Type":"ContainerStarted","Data":"7c492156ec81a6a02a0ae21316b17ad6ce6dee833f6a58f70be1f426f1b2d67d"} Aug 04 16:03:42.892804 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.892778 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t7vhm" event={"ID":"4081f9ba-ac7b-4f25-b676-4af064e18d99","Type":"ContainerStarted","Data":"bec0b9b1bae58ffd9d8e6b7ad347e0d315fea658ef8efb45c1b9ae722ed33f1f"} Aug 04 16:03:42.892862 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.892830 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t7vhm" event={"ID":"4081f9ba-ac7b-4f25-b676-4af064e18d99","Type":"ContainerStarted","Data":"162ef6b9ad91aad453e09f3ab84cca470717fef0e141d390fa4868d5345e05e9"} Aug 04 16:03:42.893734 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.893705 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf" event={"ID":"97c4bc8f-306d-41b3-bd01-16fbb9e475a7","Type":"ContainerStarted","Data":"f9b19e9bc6bb0e8e557be083d3c28b5110ee1d1e03804a764519f2f83aec7c11"} Aug 04 16:03:42.894716 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.894691 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" event={"ID":"a41c1a1a-6e8a-44d6-ae90-47b0763493be","Type":"ContainerStarted","Data":"4fc9e077b0960787b55a59bb4f6488770a6465465ecbad1c9c317578e002b841"} Aug 04 16:03:42.895635 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:42.895618 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg" event={"ID":"27fd7d60-16e2-4dcb-838c-ace287a26c21","Type":"ContainerStarted","Data":"a901dfac46130eefa4e4b4028ac5334f5535ed6b825dd7ce481fbe008de69fe4"} Aug 04 16:03:43.904944 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:43.904832 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t7vhm" event={"ID":"4081f9ba-ac7b-4f25-b676-4af064e18d99","Type":"ContainerStarted","Data":"16ab6f1e269e1a440ad27d66360fd070e0109f5a38df850a1b69edd79e58dad5"} Aug 04 16:03:43.908142 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:43.907133 2580 generic.go:356] "Generic (PLEG): container finished" podID="8e015152-d986-4b2b-9f93-296d9291b74e" containerID="c18b07e9cbb1a425ff9fee6f066d9bc3a609d06e01041f39e26b85df7f86bdea" exitCode=0 Aug 04 16:03:43.908142 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:43.907182 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-8scs4" event={"ID":"8e015152-d986-4b2b-9f93-296d9291b74e","Type":"ContainerDied","Data":"c18b07e9cbb1a425ff9fee6f066d9bc3a609d06e01041f39e26b85df7f86bdea"} Aug 04 16:03:43.908142 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:43.907592 2580 scope.go:117] "RemoveContainer" containerID="c18b07e9cbb1a425ff9fee6f066d9bc3a609d06e01041f39e26b85df7f86bdea" Aug 04 16:03:44.925687 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:44.923198 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dvdq8" event={"ID":"031a5dd3-4e36-4920-b9a3-d732b8a0a0f5","Type":"ContainerStarted","Data":"751ef157a4a178e434c0b625b30c7bd5ef4339cff3eaaf94ad7f6ec29488c8c4"} Aug 04 16:03:44.925687 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:44.925632 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dvdq8" Aug 04 16:03:44.932032 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:44.931987 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dvdq8" Aug 04 16:03:44.933794 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:44.933309 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-8scs4" event={"ID":"8e015152-d986-4b2b-9f93-296d9291b74e","Type":"ContainerStarted","Data":"4996d8c7afe2a1f1ff5b8b41981982652c5d6a71da408d130d450282e5ee5096"} Aug 04 16:03:44.945410 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:44.944767 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dvdq8" podStartSLOduration=2.242871938 podStartE2EDuration="3.94474906s" totalImagesPullingTime="1.701877122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:03:44.943412978 +0000 UTC m=+126.087745250" watchObservedRunningTime="2026-08-04 16:03:44.94474906 +0000 UTC m=+126.089081307" Aug 04 16:03:45.515150 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.515121 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-tcsrs_6f235ea2-00aa-471c-b792-a2ed764869fa/networking-console-plugin/0.log" Aug 04 16:03:45.531488 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.531464 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-5sk6d"] Aug 04 16:03:45.535091 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.535069 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" Aug 04 16:03:45.539506 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.539485 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Aug 04 16:03:45.539763 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.539729 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-g5tbg\"" Aug 04 16:03:45.541642 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.541622 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Aug 04 16:03:45.541826 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.541723 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 16:03:45.547700 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.547239 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-5sk6d"] Aug 04 16:03:45.560470 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.560207 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dpzt\" (UniqueName: \"kubernetes.io/projected/5fecb59b-bad4-4214-92cf-692c79ec448a-kube-api-access-8dpzt\") pod \"prometheus-operator-758d7cb874-5sk6d\" (UID: \"5fecb59b-bad4-4214-92cf-692c79ec448a\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" Aug 04 16:03:45.560470 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.560297 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5fecb59b-bad4-4214-92cf-692c79ec448a-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-5sk6d\" (UID: \"5fecb59b-bad4-4214-92cf-692c79ec448a\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" Aug 04 16:03:45.560470 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.560345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5fecb59b-bad4-4214-92cf-692c79ec448a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-5sk6d\" (UID: \"5fecb59b-bad4-4214-92cf-692c79ec448a\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" Aug 04 16:03:45.560470 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.560385 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fecb59b-bad4-4214-92cf-692c79ec448a-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-5sk6d\" (UID: \"5fecb59b-bad4-4214-92cf-692c79ec448a\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" Aug 04 16:03:45.662305 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.660897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dpzt\" (UniqueName: \"kubernetes.io/projected/5fecb59b-bad4-4214-92cf-692c79ec448a-kube-api-access-8dpzt\") pod \"prometheus-operator-758d7cb874-5sk6d\" (UID: \"5fecb59b-bad4-4214-92cf-692c79ec448a\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" Aug 04 16:03:45.662305 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.660946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5fecb59b-bad4-4214-92cf-692c79ec448a-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-5sk6d\" (UID: \"5fecb59b-bad4-4214-92cf-692c79ec448a\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" Aug 04 16:03:45.662305 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.660986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5fecb59b-bad4-4214-92cf-692c79ec448a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-5sk6d\" (UID: \"5fecb59b-bad4-4214-92cf-692c79ec448a\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" Aug 04 16:03:45.662305 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.661023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fecb59b-bad4-4214-92cf-692c79ec448a-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-5sk6d\" (UID: \"5fecb59b-bad4-4214-92cf-692c79ec448a\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" Aug 04 16:03:45.662305 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.662004 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fecb59b-bad4-4214-92cf-692c79ec448a-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-5sk6d\" (UID: \"5fecb59b-bad4-4214-92cf-692c79ec448a\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" Aug 04 16:03:45.662305 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:03:45.662188 2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Aug 04 16:03:45.663114 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:03:45.662843 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fecb59b-bad4-4214-92cf-692c79ec448a-prometheus-operator-tls podName:5fecb59b-bad4-4214-92cf-692c79ec448a nodeName:}" failed. No retries permitted until 2026-08-04 16:03:46.162813737 +0000 UTC m=+127.307145966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/5fecb59b-bad4-4214-92cf-692c79ec448a-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-5sk6d" (UID: "5fecb59b-bad4-4214-92cf-692c79ec448a") : secret "prometheus-operator-tls" not found Aug 04 16:03:45.667312 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.667246 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5fecb59b-bad4-4214-92cf-692c79ec448a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-5sk6d\" (UID: \"5fecb59b-bad4-4214-92cf-692c79ec448a\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" Aug 04 16:03:45.675480 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.675364 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dpzt\" (UniqueName: \"kubernetes.io/projected/5fecb59b-bad4-4214-92cf-692c79ec448a-kube-api-access-8dpzt\") pod \"prometheus-operator-758d7cb874-5sk6d\" (UID: \"5fecb59b-bad4-4214-92cf-692c79ec448a\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" Aug 04 16:03:45.717210 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.717184 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:03:45.922696 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:45.922668 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/2.log" Aug 04 16:03:46.166012 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:46.165976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5fecb59b-bad4-4214-92cf-692c79ec448a-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-5sk6d\" (UID: \"5fecb59b-bad4-4214-92cf-692c79ec448a\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" Aug 04 16:03:46.172313 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:46.172241 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5fecb59b-bad4-4214-92cf-692c79ec448a-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-5sk6d\" (UID: \"5fecb59b-bad4-4214-92cf-692c79ec448a\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" Aug 04 16:03:46.451483 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:46.451231 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" Aug 04 16:03:46.716720 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:46.716597 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c7ddb6d8b-g9s7p_64e539db-01a3-48e9-bf35-35807cebc38f/registry/0.log" Aug 04 16:03:46.915164 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:46.915115 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c7ddb6d8b-g9s7p_64e539db-01a3-48e9-bf35-35807cebc38f/registry/1.log" Aug 04 16:03:47.516867 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:47.516835 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vbhgz_56b8e09d-90bb-401d-96a6-20d691168fc3/node-ca/0.log" Aug 04 16:03:47.715937 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:47.715868 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d8f46644-nmgqm_66928c11-68ad-4888-8ee8-82a21af92095/router/0.log" Aug 04 16:03:47.915482 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:47.915448 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v8jrh_a9b9081e-ce41-4a27-97c9-d02d7bd2bab7/serve-healthcheck-canary/0.log" Aug 04 16:03:52.114504 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:52.114389 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-5sk6d"] Aug 04 16:03:52.117351 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:03:52.117306 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fecb59b_bad4_4214_92cf_692c79ec448a.slice/crio-12407c5873700ee779629bc99e7fe74f5d650952322acd4a432b2b55d7b6ce5f WatchSource:0}: Error finding container 12407c5873700ee779629bc99e7fe74f5d650952322acd4a432b2b55d7b6ce5f: Status 404 returned error can't find the container with id 12407c5873700ee779629bc99e7fe74f5d650952322acd4a432b2b55d7b6ce5f Aug 04 16:03:52.977818 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:52.977758 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-577c6684f8-9sf6c" event={"ID":"684f9c20-4cdb-40c0-9350-4495bf1fe6c1","Type":"ContainerStarted","Data":"e38ee6e745196e7730f6231c789225db36dd7ff1405dc7fc3829251614346703"} Aug 04 16:03:52.980194 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:52.980156 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t7vhm" event={"ID":"4081f9ba-ac7b-4f25-b676-4af064e18d99","Type":"ContainerStarted","Data":"8281cd3a3acb2423d01bfab2afafa6b62b7b2374a1273a587e7d3f7bf4cd7369"} Aug 04 16:03:52.981451 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:52.981425 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" event={"ID":"5fecb59b-bad4-4214-92cf-692c79ec448a","Type":"ContainerStarted","Data":"12407c5873700ee779629bc99e7fe74f5d650952322acd4a432b2b55d7b6ce5f"} Aug 04 16:03:52.983156 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:52.983127 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf" event={"ID":"97c4bc8f-306d-41b3-bd01-16fbb9e475a7","Type":"ContainerStarted","Data":"ca33a5cf8b7f71af995debfaf1b10fa4b9ef9b3ebd98114d9b1fe177e104dc2c"} Aug 04 16:03:52.985381 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:52.985359 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" event={"ID":"a41c1a1a-6e8a-44d6-ae90-47b0763493be","Type":"ContainerStarted","Data":"41bf92107ffcb17d408df64bfa6c7b1b3895a4c9b6de69c13e0bcc3b1a44aa07"} Aug 04 16:03:52.985463 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:52.985388 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" event={"ID":"a41c1a1a-6e8a-44d6-ae90-47b0763493be","Type":"ContainerStarted","Data":"85e9533ef0bd30c4a629c3d11d031e1b2538adecc641613dbb48eda314306f53"} Aug 04 16:03:52.985463 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:52.985399 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" event={"ID":"a41c1a1a-6e8a-44d6-ae90-47b0763493be","Type":"ContainerStarted","Data":"f216ce1ba6b168db73903e029f5183597256594e1c80d92f421a449c0cda9316"} Aug 04 16:03:52.986878 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:52.986858 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg" event={"ID":"27fd7d60-16e2-4dcb-838c-ace287a26c21","Type":"ContainerStarted","Data":"5f22bed88aec83b3f45bb891d3015a1a2dc745731de5243032e2f4dafefa42a6"} Aug 04 16:03:52.987082 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:52.987063 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg" Aug 04 16:03:52.988891 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:52.988871 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg" Aug 04 16:03:52.997692 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:52.997655 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-577c6684f8-9sf6c" podStartSLOduration=2.509744519 podStartE2EDuration="11.997642848s" totalImagesPullingTime="9.487898329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:03:52.995465279 +0000 UTC m=+134.139797557" watchObservedRunningTime="2026-08-04 16:03:52.997642848 +0000 UTC m=+134.141975072" Aug 04 16:03:53.015240 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:53.015205 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5597d5df7-bqc8q" podStartSLOduration=2.2248119 podStartE2EDuration="12.01519423s" totalImagesPullingTime="9.79038233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:03:53.013985874 +0000 UTC m=+134.158318121" watchObservedRunningTime="2026-08-04 16:03:53.01519423 +0000 UTC m=+134.159526475" Aug 04 16:03:53.032522 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:53.032486 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686789cd78-dp8fg" podStartSLOduration=2.237398955 podStartE2EDuration="12.032472693s" totalImagesPullingTime="9.795073738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:03:53.031438402 +0000 UTC m=+134.175770659" watchObservedRunningTime="2026-08-04 16:03:53.032472693 +0000 UTC m=+134.176804938" Aug 04 16:03:53.050946 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:53.050911 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-t7vhm" podStartSLOduration=2.748940284 podStartE2EDuration="12.05089973s" totalImagesPullingTime="9.301959446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 16:03:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:03:53.050166713 +0000 UTC m=+134.194498986" watchObservedRunningTime="2026-08-04 16:03:53.05089973 +0000 UTC m=+134.195232009" Aug 04 16:03:53.068615 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:53.068573 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674cc6bcfd-l7gdf" podStartSLOduration=2.388952625 podStartE2EDuration="12.068565689s" totalImagesPullingTime="9.679613064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:03:53.067294792 +0000 UTC m=+134.211627033" watchObservedRunningTime="2026-08-04 16:03:53.068565689 +0000 UTC m=+134.212897935" Aug 04 16:03:54.863343 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:03:54.863249 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:04:02.291661 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:02.291627 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:04:02.292089 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:02.291714 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:04:02.296482 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:02.296455 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:04:03.044166 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:03.044147 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:04:04.045983 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:04.045944 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-kw7q7" event={"ID":"41a9698f-7fff-4a1e-915a-bb1875387c14","Type":"ContainerStarted","Data":"ea7d00f7a129d50ee9827dc0fc4296b4d288582c655843d8a1d14c147ed29601"} Aug 04 16:04:04.046800 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:04.046774 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-kw7q7" Aug 04 16:04:04.048562 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:04.048538 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" event={"ID":"5fecb59b-bad4-4214-92cf-692c79ec448a","Type":"ContainerStarted","Data":"6dca247645063ebbb3e4b6e8e68852bacf15b94191f68d24b0656c52d47cf634"} Aug 04 16:04:04.048743 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:04.048716 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" event={"ID":"5fecb59b-bad4-4214-92cf-692c79ec448a","Type":"ContainerStarted","Data":"044ca8c1d10992fd7a5476f4be1d8a09de91b2a1f21db97628bd5861bccc797e"} Aug 04 16:04:04.052792 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:04.052767 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-kw7q7" Aug 04 16:04:04.068743 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:04.068689 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-kw7q7" podStartSLOduration=2.463621912 podStartE2EDuration="23.068670648s" totalImagesPullingTime="20.605048736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:04:04.067229419 +0000 UTC m=+145.211561703" watchObservedRunningTime="2026-08-04 16:04:04.068670648 +0000 UTC m=+145.213002895" Aug 04 16:04:04.105351 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:04.105275 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-5sk6d" podStartSLOduration=8.205545921 podStartE2EDuration="19.105257059s" totalImagesPullingTime="10.899711138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:04:04.104641617 +0000 UTC m=+145.248973886" watchObservedRunningTime="2026-08-04 16:04:04.105257059 +0000 UTC m=+145.249589307" Aug 04 16:04:05.913781 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:05.913735 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hrf2c"] Aug 04 16:04:05.951147 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:05.951122 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:05.955819 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:05.954961 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 16:04:05.955819 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:05.955190 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 16:04:05.955819 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:05.955441 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 16:04:05.955819 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:05.955642 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j6txb\"" Aug 04 16:04:06.038612 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.038177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngqvw\" (UniqueName: \"kubernetes.io/projected/2a8f86c0-08b1-47e6-be77-05099a719954-kube-api-access-ngqvw\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.038612 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.038224 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a8f86c0-08b1-47e6-be77-05099a719954-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.038612 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.038258 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a8f86c0-08b1-47e6-be77-05099a719954-node-exporter-textfile\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.038612 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.038321 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a8f86c0-08b1-47e6-be77-05099a719954-sys\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.038612 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.038359 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a8f86c0-08b1-47e6-be77-05099a719954-metrics-client-ca\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.038612 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.038392 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a8f86c0-08b1-47e6-be77-05099a719954-node-exporter-wtmp\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.038612 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.038417 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a8f86c0-08b1-47e6-be77-05099a719954-node-exporter-accelerators-collector-config\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.038612 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.038463 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a8f86c0-08b1-47e6-be77-05099a719954-root\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.038612 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.038486 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a8f86c0-08b1-47e6-be77-05099a719954-node-exporter-tls\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.139162 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.139122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a8f86c0-08b1-47e6-be77-05099a719954-node-exporter-wtmp\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.139162 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.139172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a8f86c0-08b1-47e6-be77-05099a719954-node-exporter-accelerators-collector-config\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.139419 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.139353 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a8f86c0-08b1-47e6-be77-05099a719954-node-exporter-wtmp\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.139481 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.139415 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a8f86c0-08b1-47e6-be77-05099a719954-root\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.139481 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.139451 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a8f86c0-08b1-47e6-be77-05099a719954-node-exporter-tls\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.139578 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.139491 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqvw\" (UniqueName: \"kubernetes.io/projected/2a8f86c0-08b1-47e6-be77-05099a719954-kube-api-access-ngqvw\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.139578 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.139526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a8f86c0-08b1-47e6-be77-05099a719954-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.139578 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.139558 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a8f86c0-08b1-47e6-be77-05099a719954-node-exporter-textfile\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.139723 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.139613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a8f86c0-08b1-47e6-be77-05099a719954-sys\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.139723 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.139649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a8f86c0-08b1-47e6-be77-05099a719954-metrics-client-ca\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.139818 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.139742 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a8f86c0-08b1-47e6-be77-05099a719954-node-exporter-accelerators-collector-config\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.140177 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.140153 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a8f86c0-08b1-47e6-be77-05099a719954-metrics-client-ca\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.140420 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.140399 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a8f86c0-08b1-47e6-be77-05099a719954-node-exporter-textfile\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.140744 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.140719 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a8f86c0-08b1-47e6-be77-05099a719954-sys\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.141352 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.141329 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a8f86c0-08b1-47e6-be77-05099a719954-root\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.142461 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.142437 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a8f86c0-08b1-47e6-be77-05099a719954-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.143435 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.143415 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a8f86c0-08b1-47e6-be77-05099a719954-node-exporter-tls\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.152101 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.152082 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngqvw\" (UniqueName: \"kubernetes.io/projected/2a8f86c0-08b1-47e6-be77-05099a719954-kube-api-access-ngqvw\") pod \"node-exporter-hrf2c\" (UID: \"2a8f86c0-08b1-47e6-be77-05099a719954\") " pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:06.266774 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:06.266695 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hrf2c" Aug 04 16:04:07.070201 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:07.070160 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hrf2c" event={"ID":"2a8f86c0-08b1-47e6-be77-05099a719954","Type":"ContainerStarted","Data":"ed0e5784a7265340c6e678ae680c6d7e88343f68f57973dec47e58e065824a7b"} Aug 04 16:04:09.082272 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:09.082235 2580 generic.go:356] "Generic (PLEG): container finished" podID="2a8f86c0-08b1-47e6-be77-05099a719954" containerID="4696d1afe13c6b7b73127637b6bca23de236c947bfeaa329630ab087a566efed" exitCode=0 Aug 04 16:04:09.082816 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:09.082407 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hrf2c" event={"ID":"2a8f86c0-08b1-47e6-be77-05099a719954","Type":"ContainerDied","Data":"4696d1afe13c6b7b73127637b6bca23de236c947bfeaa329630ab087a566efed"} Aug 04 16:04:10.089527 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:10.089482 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hrf2c" event={"ID":"2a8f86c0-08b1-47e6-be77-05099a719954","Type":"ContainerStarted","Data":"5e98f2616330df8e2664abf4e44e91196410f3816340165e2732555ada517b63"} Aug 04 16:04:10.089921 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:10.089589 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hrf2c" event={"ID":"2a8f86c0-08b1-47e6-be77-05099a719954","Type":"ContainerStarted","Data":"e1cb853c87bf98ab7b1d28d1f20c0e6dbb0a8a98ff0cb1c52b14de4fdc882a0e"} Aug 04 16:04:10.120532 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:10.120484 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hrf2c" podStartSLOduration=3.282836339 podStartE2EDuration="5.120468573s" totalImagesPullingTime="1.744663259s" totalInitContainerRuntime="92.968975ms" isStatefulPod=true podCreationTimestamp="2026-08-04 16:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:04:10.11768107 +0000 UTC m=+151.262013339" watchObservedRunningTime="2026-08-04 16:04:10.120468573 +0000 UTC m=+151.264800819" Aug 04 16:04:10.124055 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:10.124026 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p"] Aug 04 16:04:21.262550 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:21.262514 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-577c6684f8-9sf6c"] Aug 04 16:04:35.145466 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.145364 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" containerName="registry" containerID="cri-o://d7fdab947f24fd826b8b083bfb936a2e57a7e51ecebfa010c84b5cb995433db1" gracePeriod=30 Aug 04 16:04:35.413073 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.413005 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:04:35.580550 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.580517 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64e539db-01a3-48e9-bf35-35807cebc38f-trusted-ca\") pod \"64e539db-01a3-48e9-bf35-35807cebc38f\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " Aug 04 16:04:35.580731 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.580580 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-bound-sa-token\") pod \"64e539db-01a3-48e9-bf35-35807cebc38f\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " Aug 04 16:04:35.580731 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.580630 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64e539db-01a3-48e9-bf35-35807cebc38f-installation-pull-secrets\") pod \"64e539db-01a3-48e9-bf35-35807cebc38f\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " Aug 04 16:04:35.580731 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.580656 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvrks\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-kube-api-access-hvrks\") pod \"64e539db-01a3-48e9-bf35-35807cebc38f\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " Aug 04 16:04:35.580731 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.580692 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls\") pod \"64e539db-01a3-48e9-bf35-35807cebc38f\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " Aug 04 16:04:35.580931 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.580741 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64e539db-01a3-48e9-bf35-35807cebc38f-registry-certificates\") pod \"64e539db-01a3-48e9-bf35-35807cebc38f\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " Aug 04 16:04:35.580931 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.580782 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64e539db-01a3-48e9-bf35-35807cebc38f-image-registry-private-configuration\") pod \"64e539db-01a3-48e9-bf35-35807cebc38f\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " Aug 04 16:04:35.580931 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.580809 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64e539db-01a3-48e9-bf35-35807cebc38f-ca-trust-extracted\") pod \"64e539db-01a3-48e9-bf35-35807cebc38f\" (UID: \"64e539db-01a3-48e9-bf35-35807cebc38f\") " Aug 04 16:04:35.580931 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.580902 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e539db-01a3-48e9-bf35-35807cebc38f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "64e539db-01a3-48e9-bf35-35807cebc38f" (UID: "64e539db-01a3-48e9-bf35-35807cebc38f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:04:35.581219 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.581187 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64e539db-01a3-48e9-bf35-35807cebc38f-trusted-ca\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:04:35.581393 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.581228 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e539db-01a3-48e9-bf35-35807cebc38f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "64e539db-01a3-48e9-bf35-35807cebc38f" (UID: "64e539db-01a3-48e9-bf35-35807cebc38f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:04:35.583218 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.583192 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-kube-api-access-hvrks" (OuterVolumeSpecName: "kube-api-access-hvrks") pod "64e539db-01a3-48e9-bf35-35807cebc38f" (UID: "64e539db-01a3-48e9-bf35-35807cebc38f"). InnerVolumeSpecName "kube-api-access-hvrks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:04:35.583345 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.583307 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e539db-01a3-48e9-bf35-35807cebc38f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "64e539db-01a3-48e9-bf35-35807cebc38f" (UID: "64e539db-01a3-48e9-bf35-35807cebc38f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:04:35.583345 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.583326 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "64e539db-01a3-48e9-bf35-35807cebc38f" (UID: "64e539db-01a3-48e9-bf35-35807cebc38f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:04:35.583518 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.583400 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e539db-01a3-48e9-bf35-35807cebc38f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "64e539db-01a3-48e9-bf35-35807cebc38f" (UID: "64e539db-01a3-48e9-bf35-35807cebc38f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:04:35.583518 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.583440 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "64e539db-01a3-48e9-bf35-35807cebc38f" (UID: "64e539db-01a3-48e9-bf35-35807cebc38f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:04:35.589198 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.589173 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e539db-01a3-48e9-bf35-35807cebc38f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "64e539db-01a3-48e9-bf35-35807cebc38f" (UID: "64e539db-01a3-48e9-bf35-35807cebc38f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:04:35.681645 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.681588 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-registry-tls\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:04:35.681645 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.681611 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64e539db-01a3-48e9-bf35-35807cebc38f-registry-certificates\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:04:35.681645 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.681622 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64e539db-01a3-48e9-bf35-35807cebc38f-image-registry-private-configuration\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:04:35.681645 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.681631 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64e539db-01a3-48e9-bf35-35807cebc38f-ca-trust-extracted\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:04:35.681645 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.681639 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-bound-sa-token\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:04:35.681645 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.681647 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64e539db-01a3-48e9-bf35-35807cebc38f-installation-pull-secrets\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:04:35.681889 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:35.681655 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvrks\" (UniqueName: \"kubernetes.io/projected/64e539db-01a3-48e9-bf35-35807cebc38f-kube-api-access-hvrks\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:04:36.210019 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:36.209986 2580 generic.go:356] "Generic (PLEG): container finished" podID="64e539db-01a3-48e9-bf35-35807cebc38f" containerID="d7fdab947f24fd826b8b083bfb936a2e57a7e51ecebfa010c84b5cb995433db1" exitCode=0 Aug 04 16:04:36.210450 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:36.210034 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" Aug 04 16:04:36.210450 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:36.210051 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" event={"ID":"64e539db-01a3-48e9-bf35-35807cebc38f","Type":"ContainerDied","Data":"d7fdab947f24fd826b8b083bfb936a2e57a7e51ecebfa010c84b5cb995433db1"} Aug 04 16:04:36.210450 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:36.210117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p" event={"ID":"64e539db-01a3-48e9-bf35-35807cebc38f","Type":"ContainerDied","Data":"78401696b1b5fc29de7fda001fe73e78199c1140d1c1b07bfb045f635e7dfb9c"} Aug 04 16:04:36.210450 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:36.210136 2580 scope.go:117] "RemoveContainer" containerID="d7fdab947f24fd826b8b083bfb936a2e57a7e51ecebfa010c84b5cb995433db1" Aug 04 16:04:36.219110 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:36.219089 2580 scope.go:117] "RemoveContainer" containerID="f27a55a7ad6a82964a52223efe7a56a04f42372b5db4c1637e4de4c79e4cbc7a" Aug 04 16:04:36.226325 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:36.226307 2580 scope.go:117] "RemoveContainer" containerID="d7fdab947f24fd826b8b083bfb936a2e57a7e51ecebfa010c84b5cb995433db1" Aug 04 16:04:36.226584 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:04:36.226553 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7fdab947f24fd826b8b083bfb936a2e57a7e51ecebfa010c84b5cb995433db1\": container with ID starting with d7fdab947f24fd826b8b083bfb936a2e57a7e51ecebfa010c84b5cb995433db1 not found: ID does not exist" containerID="d7fdab947f24fd826b8b083bfb936a2e57a7e51ecebfa010c84b5cb995433db1" Aug 04 16:04:36.226656 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:36.226597 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7fdab947f24fd826b8b083bfb936a2e57a7e51ecebfa010c84b5cb995433db1"} err="failed to get container status \"d7fdab947f24fd826b8b083bfb936a2e57a7e51ecebfa010c84b5cb995433db1\": rpc error: code = NotFound desc = could not find container \"d7fdab947f24fd826b8b083bfb936a2e57a7e51ecebfa010c84b5cb995433db1\": container with ID starting with d7fdab947f24fd826b8b083bfb936a2e57a7e51ecebfa010c84b5cb995433db1 not found: ID does not exist" Aug 04 16:04:36.226656 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:36.226620 2580 scope.go:117] "RemoveContainer" containerID="f27a55a7ad6a82964a52223efe7a56a04f42372b5db4c1637e4de4c79e4cbc7a" Aug 04 16:04:36.226878 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:04:36.226859 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27a55a7ad6a82964a52223efe7a56a04f42372b5db4c1637e4de4c79e4cbc7a\": container with ID starting with f27a55a7ad6a82964a52223efe7a56a04f42372b5db4c1637e4de4c79e4cbc7a not found: ID does not exist" containerID="f27a55a7ad6a82964a52223efe7a56a04f42372b5db4c1637e4de4c79e4cbc7a" Aug 04 16:04:36.226915 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:36.226884 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27a55a7ad6a82964a52223efe7a56a04f42372b5db4c1637e4de4c79e4cbc7a"} err="failed to get container status \"f27a55a7ad6a82964a52223efe7a56a04f42372b5db4c1637e4de4c79e4cbc7a\": rpc error: code = NotFound desc = could not find container \"f27a55a7ad6a82964a52223efe7a56a04f42372b5db4c1637e4de4c79e4cbc7a\": container with ID starting with f27a55a7ad6a82964a52223efe7a56a04f42372b5db4c1637e4de4c79e4cbc7a not found: ID does not exist" Aug 04 16:04:36.231606 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:36.231586 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p"] Aug 04 16:04:36.234512 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:36.234492 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c7ddb6d8b-g9s7p"] Aug 04 16:04:37.318481 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:37.318450 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" path="/var/lib/kubelet/pods/64e539db-01a3-48e9-bf35-35807cebc38f/volumes" Aug 04 16:04:46.281438 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.281380 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-577c6684f8-9sf6c" podUID="684f9c20-4cdb-40c0-9350-4495bf1fe6c1" containerName="console" containerID="cri-o://e38ee6e745196e7730f6231c789225db36dd7ff1405dc7fc3829251614346703" gracePeriod=15 Aug 04 16:04:46.552821 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.552771 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:04:46.559886 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.559868 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64jxx\" (UniqueName: \"kubernetes.io/projected/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-kube-api-access-64jxx\") pod \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " Aug 04 16:04:46.559961 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.559899 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-oauth-serving-cert\") pod \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " Aug 04 16:04:46.559961 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.559918 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-serving-cert\") pod \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " Aug 04 16:04:46.559961 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.559936 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-trusted-ca-bundle\") pod \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " Aug 04 16:04:46.560079 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.559961 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-config\") pod \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " Aug 04 16:04:46.560079 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.559998 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-service-ca\") pod \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " Aug 04 16:04:46.560079 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.560027 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-oauth-config\") pod \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\" (UID: \"684f9c20-4cdb-40c0-9350-4495bf1fe6c1\") " Aug 04 16:04:46.560369 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.560336 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "684f9c20-4cdb-40c0-9350-4495bf1fe6c1" (UID: "684f9c20-4cdb-40c0-9350-4495bf1fe6c1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:04:46.560449 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.560359 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "684f9c20-4cdb-40c0-9350-4495bf1fe6c1" (UID: "684f9c20-4cdb-40c0-9350-4495bf1fe6c1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:04:46.560449 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.560391 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-config" (OuterVolumeSpecName: "console-config") pod "684f9c20-4cdb-40c0-9350-4495bf1fe6c1" (UID: "684f9c20-4cdb-40c0-9350-4495bf1fe6c1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:04:46.560449 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.560417 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-service-ca" (OuterVolumeSpecName: "service-ca") pod "684f9c20-4cdb-40c0-9350-4495bf1fe6c1" (UID: "684f9c20-4cdb-40c0-9350-4495bf1fe6c1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:04:46.562213 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.562192 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "684f9c20-4cdb-40c0-9350-4495bf1fe6c1" (UID: "684f9c20-4cdb-40c0-9350-4495bf1fe6c1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:04:46.562486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.562465 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "684f9c20-4cdb-40c0-9350-4495bf1fe6c1" (UID: "684f9c20-4cdb-40c0-9350-4495bf1fe6c1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:04:46.562486 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.562473 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-kube-api-access-64jxx" (OuterVolumeSpecName: "kube-api-access-64jxx") pod "684f9c20-4cdb-40c0-9350-4495bf1fe6c1" (UID: "684f9c20-4cdb-40c0-9350-4495bf1fe6c1"). InnerVolumeSpecName "kube-api-access-64jxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:04:46.661557 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.661534 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-service-ca\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:04:46.661557 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.661557 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-oauth-config\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:04:46.661557 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.661566 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64jxx\" (UniqueName: \"kubernetes.io/projected/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-kube-api-access-64jxx\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:04:46.661733 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.661574 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-oauth-serving-cert\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:04:46.661733 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.661584 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-serving-cert\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:04:46.661733 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.661592 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-trusted-ca-bundle\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:04:46.661733 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:46.661600 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/684f9c20-4cdb-40c0-9350-4495bf1fe6c1-console-config\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:04:47.256450 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:47.256416 2580 generic.go:356] "Generic (PLEG): container finished" podID="684f9c20-4cdb-40c0-9350-4495bf1fe6c1" containerID="e38ee6e745196e7730f6231c789225db36dd7ff1405dc7fc3829251614346703" exitCode=0 Aug 04 16:04:47.256600 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:47.256461 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-577c6684f8-9sf6c" Aug 04 16:04:47.256600 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:47.256467 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-577c6684f8-9sf6c" event={"ID":"684f9c20-4cdb-40c0-9350-4495bf1fe6c1","Type":"ContainerDied","Data":"e38ee6e745196e7730f6231c789225db36dd7ff1405dc7fc3829251614346703"} Aug 04 16:04:47.256600 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:47.256516 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-577c6684f8-9sf6c" event={"ID":"684f9c20-4cdb-40c0-9350-4495bf1fe6c1","Type":"ContainerDied","Data":"09598eaaf3050ddf9fb8fdac291c97f30f6471e89e6ea44e60db5eed0dc57d08"} Aug 04 16:04:47.256600 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:47.256542 2580 scope.go:117] "RemoveContainer" containerID="e38ee6e745196e7730f6231c789225db36dd7ff1405dc7fc3829251614346703" Aug 04 16:04:47.265078 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:47.265052 2580 scope.go:117] "RemoveContainer" containerID="e38ee6e745196e7730f6231c789225db36dd7ff1405dc7fc3829251614346703" Aug 04 16:04:47.265353 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:04:47.265333 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38ee6e745196e7730f6231c789225db36dd7ff1405dc7fc3829251614346703\": container with ID starting with e38ee6e745196e7730f6231c789225db36dd7ff1405dc7fc3829251614346703 not found: ID does not exist" containerID="e38ee6e745196e7730f6231c789225db36dd7ff1405dc7fc3829251614346703" Aug 04 16:04:47.265442 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:47.265358 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38ee6e745196e7730f6231c789225db36dd7ff1405dc7fc3829251614346703"} err="failed to get container status \"e38ee6e745196e7730f6231c789225db36dd7ff1405dc7fc3829251614346703\": rpc error: code = NotFound desc = could not find container \"e38ee6e745196e7730f6231c789225db36dd7ff1405dc7fc3829251614346703\": container with ID starting with e38ee6e745196e7730f6231c789225db36dd7ff1405dc7fc3829251614346703 not found: ID does not exist" Aug 04 16:04:47.278513 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:47.278490 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-577c6684f8-9sf6c"] Aug 04 16:04:47.282624 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:47.282603 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-577c6684f8-9sf6c"] Aug 04 16:04:47.318458 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:04:47.318437 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="684f9c20-4cdb-40c0-9350-4495bf1fe6c1" path="/var/lib/kubelet/pods/684f9c20-4cdb-40c0-9350-4495bf1fe6c1/volumes" Aug 04 16:06:39.184406 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:06:39.184379 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:06:39.190669 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:06:39.190642 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:06:39.195019 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:06:39.194999 2580 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 16:08:43.329100 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.329069 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j"] Aug 04 16:08:43.329544 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.329381 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="684f9c20-4cdb-40c0-9350-4495bf1fe6c1" containerName="console" Aug 04 16:08:43.329544 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.329392 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="684f9c20-4cdb-40c0-9350-4495bf1fe6c1" containerName="console" Aug 04 16:08:43.329544 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.329399 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" containerName="registry" Aug 04 16:08:43.329544 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.329404 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" containerName="registry" Aug 04 16:08:43.329544 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.329459 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" containerName="registry" Aug 04 16:08:43.329544 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.329470 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="684f9c20-4cdb-40c0-9350-4495bf1fe6c1" containerName="console" Aug 04 16:08:43.329544 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.329529 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" containerName="registry" Aug 04 16:08:43.329544 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.329535 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" containerName="registry" Aug 04 16:08:43.329780 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.329581 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="64e539db-01a3-48e9-bf35-35807cebc38f" containerName="registry" Aug 04 16:08:43.331446 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.331430 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.334210 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.334178 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 16:08:43.334379 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.334213 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 16:08:43.334379 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.334222 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 16:08:43.334379 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.334264 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-m5dcn\"" Aug 04 16:08:43.334379 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.334266 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 16:08:43.352163 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.352140 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j"] Aug 04 16:08:43.431202 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.431181 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/031e04df-7495-43ec-b8cf-0a5f9d4bcaa9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dd47f799b-7qs4j\" (UID: \"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.431313 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.431224 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/031e04df-7495-43ec-b8cf-0a5f9d4bcaa9-tmp\") pod \"opendatahub-operator-controller-manager-7dd47f799b-7qs4j\" (UID: \"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.431313 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.431295 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/031e04df-7495-43ec-b8cf-0a5f9d4bcaa9-manifests\") pod \"opendatahub-operator-controller-manager-7dd47f799b-7qs4j\" (UID: \"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.431387 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.431345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/031e04df-7495-43ec-b8cf-0a5f9d4bcaa9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dd47f799b-7qs4j\" (UID: \"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.431387 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.431369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8scvv\" (UniqueName: \"kubernetes.io/projected/031e04df-7495-43ec-b8cf-0a5f9d4bcaa9-kube-api-access-8scvv\") pod \"opendatahub-operator-controller-manager-7dd47f799b-7qs4j\" (UID: \"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.532353 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.532327 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/031e04df-7495-43ec-b8cf-0a5f9d4bcaa9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dd47f799b-7qs4j\" (UID: \"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.532453 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.532394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/031e04df-7495-43ec-b8cf-0a5f9d4bcaa9-tmp\") pod \"opendatahub-operator-controller-manager-7dd47f799b-7qs4j\" (UID: \"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.532453 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.532440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/031e04df-7495-43ec-b8cf-0a5f9d4bcaa9-manifests\") pod \"opendatahub-operator-controller-manager-7dd47f799b-7qs4j\" (UID: \"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.532558 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.532465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/031e04df-7495-43ec-b8cf-0a5f9d4bcaa9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dd47f799b-7qs4j\" (UID: \"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.532614 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.532576 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8scvv\" (UniqueName: \"kubernetes.io/projected/031e04df-7495-43ec-b8cf-0a5f9d4bcaa9-kube-api-access-8scvv\") pod \"opendatahub-operator-controller-manager-7dd47f799b-7qs4j\" (UID: \"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.532792 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.532766 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/031e04df-7495-43ec-b8cf-0a5f9d4bcaa9-manifests\") pod \"opendatahub-operator-controller-manager-7dd47f799b-7qs4j\" (UID: \"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.534541 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.534518 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/031e04df-7495-43ec-b8cf-0a5f9d4bcaa9-tmp\") pod \"opendatahub-operator-controller-manager-7dd47f799b-7qs4j\" (UID: \"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.534710 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.534692 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/031e04df-7495-43ec-b8cf-0a5f9d4bcaa9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dd47f799b-7qs4j\" (UID: \"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.534838 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.534817 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/031e04df-7495-43ec-b8cf-0a5f9d4bcaa9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dd47f799b-7qs4j\" (UID: \"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.542434 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.542412 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8scvv\" (UniqueName: \"kubernetes.io/projected/031e04df-7495-43ec-b8cf-0a5f9d4bcaa9-kube-api-access-8scvv\") pod \"opendatahub-operator-controller-manager-7dd47f799b-7qs4j\" (UID: \"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.641615 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.641563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:43.787555 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:08:43.787529 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031e04df_7495_43ec_b8cf_0a5f9d4bcaa9.slice/crio-b97a041fcab452a86baa46620d174f31740a9e9bc8432ee1989c51ef7ed99792 WatchSource:0}: Error finding container b97a041fcab452a86baa46620d174f31740a9e9bc8432ee1989c51ef7ed99792: Status 404 returned error can't find the container with id b97a041fcab452a86baa46620d174f31740a9e9bc8432ee1989c51ef7ed99792 Aug 04 16:08:43.789146 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.789132 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:08:43.802932 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:43.802909 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j"] Aug 04 16:08:44.153022 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:44.152990 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" event={"ID":"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9","Type":"ContainerStarted","Data":"b97a041fcab452a86baa46620d174f31740a9e9bc8432ee1989c51ef7ed99792"} Aug 04 16:08:47.168087 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:47.168057 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" event={"ID":"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9","Type":"ContainerStarted","Data":"38769900ab407ed229e60add4fa1c059df6d7c13d4e677290253077ad92e7a93"} Aug 04 16:08:48.174774 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:48.174744 2580 generic.go:356] "Generic (PLEG): container finished" podID="031e04df-7495-43ec-b8cf-0a5f9d4bcaa9" containerID="38769900ab407ed229e60add4fa1c059df6d7c13d4e677290253077ad92e7a93" exitCode=0 Aug 04 16:08:48.175121 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:48.174784 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" event={"ID":"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9","Type":"ContainerDied","Data":"38769900ab407ed229e60add4fa1c059df6d7c13d4e677290253077ad92e7a93"} Aug 04 16:08:49.182649 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:49.182615 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" event={"ID":"031e04df-7495-43ec-b8cf-0a5f9d4bcaa9","Type":"ContainerStarted","Data":"796a760c935ba1792c63194629e11e8a4a0cb3fd3165b321ff460865da5620b5"} Aug 04 16:08:49.183228 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:49.182752 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:08:49.226092 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:08:49.226050 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" podStartSLOduration=1.929977741 podStartE2EDuration="6.226036552s" totalImagesPullingTime="3.342402346s" totalInitContainerRuntime="953.656465ms" isStatefulPod=false podCreationTimestamp="2026-08-04 16:08:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:08:49.224595221 +0000 UTC m=+430.368927485" watchObservedRunningTime="2026-08-04 16:08:49.226036552 +0000 UTC m=+430.370368796" Aug 04 16:09:00.188483 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:00.188453 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7dd47f799b-7qs4j" Aug 04 16:09:13.718675 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.718598 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw"] Aug 04 16:09:13.723419 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.723400 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" Aug 04 16:09:13.728247 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.728226 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 04 16:09:13.728366 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.728266 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 04 16:09:13.728366 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.728321 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 16:09:13.728366 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.728324 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 04 16:09:13.728501 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.728396 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 16:09:13.728595 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.728580 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8n2cc\"" Aug 04 16:09:13.746163 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.746133 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw"] Aug 04 16:09:13.871729 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.871699 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8b1567f-abca-48a2-9b48-0159fa23c5c1-cert\") pod \"lws-controller-manager-7c58867685-x8zbw\" (UID: \"e8b1567f-abca-48a2-9b48-0159fa23c5c1\") " pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" Aug 04 16:09:13.871867 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.871747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8b1567f-abca-48a2-9b48-0159fa23c5c1-metrics-cert\") pod \"lws-controller-manager-7c58867685-x8zbw\" (UID: \"e8b1567f-abca-48a2-9b48-0159fa23c5c1\") " pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" Aug 04 16:09:13.871867 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.871778 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flhhx\" (UniqueName: \"kubernetes.io/projected/e8b1567f-abca-48a2-9b48-0159fa23c5c1-kube-api-access-flhhx\") pod \"lws-controller-manager-7c58867685-x8zbw\" (UID: \"e8b1567f-abca-48a2-9b48-0159fa23c5c1\") " pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" Aug 04 16:09:13.871867 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.871807 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e8b1567f-abca-48a2-9b48-0159fa23c5c1-manager-config\") pod \"lws-controller-manager-7c58867685-x8zbw\" (UID: \"e8b1567f-abca-48a2-9b48-0159fa23c5c1\") " pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" Aug 04 16:09:13.972984 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.972918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8b1567f-abca-48a2-9b48-0159fa23c5c1-cert\") pod \"lws-controller-manager-7c58867685-x8zbw\" (UID: \"e8b1567f-abca-48a2-9b48-0159fa23c5c1\") " pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" Aug 04 16:09:13.972984 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.972960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8b1567f-abca-48a2-9b48-0159fa23c5c1-metrics-cert\") pod \"lws-controller-manager-7c58867685-x8zbw\" (UID: \"e8b1567f-abca-48a2-9b48-0159fa23c5c1\") " pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" Aug 04 16:09:13.972984 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.972981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flhhx\" (UniqueName: \"kubernetes.io/projected/e8b1567f-abca-48a2-9b48-0159fa23c5c1-kube-api-access-flhhx\") pod \"lws-controller-manager-7c58867685-x8zbw\" (UID: \"e8b1567f-abca-48a2-9b48-0159fa23c5c1\") " pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" Aug 04 16:09:13.973197 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.973015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e8b1567f-abca-48a2-9b48-0159fa23c5c1-manager-config\") pod \"lws-controller-manager-7c58867685-x8zbw\" (UID: \"e8b1567f-abca-48a2-9b48-0159fa23c5c1\") " pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" Aug 04 16:09:13.973729 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.973711 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e8b1567f-abca-48a2-9b48-0159fa23c5c1-manager-config\") pod \"lws-controller-manager-7c58867685-x8zbw\" (UID: \"e8b1567f-abca-48a2-9b48-0159fa23c5c1\") " pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" Aug 04 16:09:13.975473 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.975443 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8b1567f-abca-48a2-9b48-0159fa23c5c1-metrics-cert\") pod \"lws-controller-manager-7c58867685-x8zbw\" (UID: \"e8b1567f-abca-48a2-9b48-0159fa23c5c1\") " pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" Aug 04 16:09:13.975564 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.975452 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8b1567f-abca-48a2-9b48-0159fa23c5c1-cert\") pod \"lws-controller-manager-7c58867685-x8zbw\" (UID: \"e8b1567f-abca-48a2-9b48-0159fa23c5c1\") " pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" Aug 04 16:09:13.990249 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:13.990228 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flhhx\" (UniqueName: \"kubernetes.io/projected/e8b1567f-abca-48a2-9b48-0159fa23c5c1-kube-api-access-flhhx\") pod \"lws-controller-manager-7c58867685-x8zbw\" (UID: \"e8b1567f-abca-48a2-9b48-0159fa23c5c1\") " pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" Aug 04 16:09:14.032814 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:14.032788 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" Aug 04 16:09:14.189913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:14.189892 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw"] Aug 04 16:09:14.192047 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:09:14.192018 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b1567f_abca_48a2_9b48_0159fa23c5c1.slice/crio-52bdc2aa9af386d00ef208ee10159b9ea5c84d44f21c0650b29ec63670287ecc WatchSource:0}: Error finding container 52bdc2aa9af386d00ef208ee10159b9ea5c84d44f21c0650b29ec63670287ecc: Status 404 returned error can't find the container with id 52bdc2aa9af386d00ef208ee10159b9ea5c84d44f21c0650b29ec63670287ecc Aug 04 16:09:14.299462 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:14.299406 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" event={"ID":"e8b1567f-abca-48a2-9b48-0159fa23c5c1","Type":"ContainerStarted","Data":"52bdc2aa9af386d00ef208ee10159b9ea5c84d44f21c0650b29ec63670287ecc"} Aug 04 16:09:15.079453 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:15.079422 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b"] Aug 04 16:09:15.084270 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:15.084250 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b" Aug 04 16:09:15.087757 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:15.087734 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Aug 04 16:09:15.087846 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:15.087826 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-6wnfz\"" Aug 04 16:09:15.107724 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:15.107703 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b"] Aug 04 16:09:15.182923 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:15.182896 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs64r\" (UniqueName: \"kubernetes.io/projected/a40745ad-c088-43d9-bf82-2cbf81fb6b09-kube-api-access-cs64r\") pod \"llmisvc-controller-manager-777c6bd7d5-h2b6b\" (UID: \"a40745ad-c088-43d9-bf82-2cbf81fb6b09\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b" Aug 04 16:09:15.183026 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:15.182952 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a40745ad-c088-43d9-bf82-2cbf81fb6b09-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h2b6b\" (UID: \"a40745ad-c088-43d9-bf82-2cbf81fb6b09\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b" Aug 04 16:09:15.284001 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:15.283885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a40745ad-c088-43d9-bf82-2cbf81fb6b09-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h2b6b\" (UID: \"a40745ad-c088-43d9-bf82-2cbf81fb6b09\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b" Aug 04 16:09:15.284001 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:15.283982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs64r\" (UniqueName: \"kubernetes.io/projected/a40745ad-c088-43d9-bf82-2cbf81fb6b09-kube-api-access-cs64r\") pod \"llmisvc-controller-manager-777c6bd7d5-h2b6b\" (UID: \"a40745ad-c088-43d9-bf82-2cbf81fb6b09\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b" Aug 04 16:09:15.284302 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:09:15.284072 2580 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Aug 04 16:09:15.284302 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:09:15.284170 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a40745ad-c088-43d9-bf82-2cbf81fb6b09-cert podName:a40745ad-c088-43d9-bf82-2cbf81fb6b09 nodeName:}" failed. No retries permitted until 2026-08-04 16:09:15.784120915 +0000 UTC m=+456.928453145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a40745ad-c088-43d9-bf82-2cbf81fb6b09-cert") pod "llmisvc-controller-manager-777c6bd7d5-h2b6b" (UID: "a40745ad-c088-43d9-bf82-2cbf81fb6b09") : secret "llmisvc-webhook-server-cert" not found Aug 04 16:09:15.304927 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:15.304879 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs64r\" (UniqueName: \"kubernetes.io/projected/a40745ad-c088-43d9-bf82-2cbf81fb6b09-kube-api-access-cs64r\") pod \"llmisvc-controller-manager-777c6bd7d5-h2b6b\" (UID: \"a40745ad-c088-43d9-bf82-2cbf81fb6b09\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b" Aug 04 16:09:15.790030 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:15.789985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a40745ad-c088-43d9-bf82-2cbf81fb6b09-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h2b6b\" (UID: \"a40745ad-c088-43d9-bf82-2cbf81fb6b09\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b" Aug 04 16:09:15.790241 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:09:15.790222 2580 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Aug 04 16:09:15.790332 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:09:15.790326 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a40745ad-c088-43d9-bf82-2cbf81fb6b09-cert podName:a40745ad-c088-43d9-bf82-2cbf81fb6b09 nodeName:}" failed. No retries permitted until 2026-08-04 16:09:16.790304667 +0000 UTC m=+457.934636905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a40745ad-c088-43d9-bf82-2cbf81fb6b09-cert") pod "llmisvc-controller-manager-777c6bd7d5-h2b6b" (UID: "a40745ad-c088-43d9-bf82-2cbf81fb6b09") : secret "llmisvc-webhook-server-cert" not found Aug 04 16:09:16.798384 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:16.798344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a40745ad-c088-43d9-bf82-2cbf81fb6b09-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h2b6b\" (UID: \"a40745ad-c088-43d9-bf82-2cbf81fb6b09\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b" Aug 04 16:09:16.800577 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:16.800552 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a40745ad-c088-43d9-bf82-2cbf81fb6b09-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h2b6b\" (UID: \"a40745ad-c088-43d9-bf82-2cbf81fb6b09\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b" Aug 04 16:09:16.895678 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:16.895646 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b" Aug 04 16:09:17.282245 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:17.282226 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b"] Aug 04 16:09:17.283513 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:09:17.283486 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda40745ad_c088_43d9_bf82_2cbf81fb6b09.slice/crio-09220c57f8f3bf6fed99f069bcc690f1e5502e05af69b440f5d1a0955d3d0454 WatchSource:0}: Error finding container 09220c57f8f3bf6fed99f069bcc690f1e5502e05af69b440f5d1a0955d3d0454: Status 404 returned error can't find the container with id 09220c57f8f3bf6fed99f069bcc690f1e5502e05af69b440f5d1a0955d3d0454 Aug 04 16:09:17.317807 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:17.317780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b" event={"ID":"a40745ad-c088-43d9-bf82-2cbf81fb6b09","Type":"ContainerStarted","Data":"09220c57f8f3bf6fed99f069bcc690f1e5502e05af69b440f5d1a0955d3d0454"} Aug 04 16:09:17.317914 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:17.317831 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" Aug 04 16:09:17.317914 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:17.317851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" event={"ID":"e8b1567f-abca-48a2-9b48-0159fa23c5c1","Type":"ContainerStarted","Data":"1a24a09ddd0379f9d05779175b5ed18407603248de0c068a3ba9fe61947a76da"} Aug 04 16:09:17.358905 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:17.358855 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" podStartSLOduration=1.34427247 podStartE2EDuration="4.358836503s" totalImagesPullingTime="3.014564033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:09:17.355880265 +0000 UTC m=+458.500212537" watchObservedRunningTime="2026-08-04 16:09:17.358836503 +0000 UTC m=+458.503168749" Aug 04 16:09:22.341875 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:22.341845 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b" event={"ID":"a40745ad-c088-43d9-bf82-2cbf81fb6b09","Type":"ContainerStarted","Data":"9d5e4de2ff431cf0a9bf97a3d4f8e736a766dfab2ea9c827c5985662da05a26b"} Aug 04 16:09:22.342238 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:22.341903 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b" Aug 04 16:09:22.388351 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:22.388305 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b" podStartSLOduration=2.945410111 podStartE2EDuration="7.388289937s" totalImagesPullingTime="4.442879826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:09:22.380921481 +0000 UTC m=+463.525253747" watchObservedRunningTime="2026-08-04 16:09:22.388289937 +0000 UTC m=+463.532622184" Aug 04 16:09:28.324327 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:28.324296 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c58867685-x8zbw" Aug 04 16:09:53.348119 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:09:53.348089 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2b6b" Aug 04 16:10:19.857358 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:19.857327 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc"] Aug 04 16:10:19.859452 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:19.859436 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" Aug 04 16:10:19.862122 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:19.862094 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 16:10:19.862242 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:19.862153 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 16:10:19.863087 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:19.863069 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4nws2\"" Aug 04 16:10:19.875431 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:19.875405 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc"] Aug 04 16:10:19.974431 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:19.974406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtkkn\" (UniqueName: \"kubernetes.io/projected/bdb0fed7-1554-4d6c-b5c3-312c8c8ef643-kube-api-access-xtkkn\") pod \"limitador-operator-controller-manager-6c549b6d85-zpslc\" (UID: \"bdb0fed7-1554-4d6c-b5c3-312c8c8ef643\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" Aug 04 16:10:20.075552 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:20.075526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtkkn\" (UniqueName: \"kubernetes.io/projected/bdb0fed7-1554-4d6c-b5c3-312c8c8ef643-kube-api-access-xtkkn\") pod \"limitador-operator-controller-manager-6c549b6d85-zpslc\" (UID: \"bdb0fed7-1554-4d6c-b5c3-312c8c8ef643\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" Aug 04 16:10:20.087488 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:20.087456 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtkkn\" (UniqueName: \"kubernetes.io/projected/bdb0fed7-1554-4d6c-b5c3-312c8c8ef643-kube-api-access-xtkkn\") pod \"limitador-operator-controller-manager-6c549b6d85-zpslc\" (UID: \"bdb0fed7-1554-4d6c-b5c3-312c8c8ef643\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" Aug 04 16:10:20.169548 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:20.169489 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" Aug 04 16:10:20.319592 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:20.319543 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc"] Aug 04 16:10:20.323187 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:10:20.323162 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb0fed7_1554_4d6c_b5c3_312c8c8ef643.slice/crio-da01c74992314b1f77bee7e5dbd40628d263c5109009166fdfada946bc65109a WatchSource:0}: Error finding container da01c74992314b1f77bee7e5dbd40628d263c5109009166fdfada946bc65109a: Status 404 returned error can't find the container with id da01c74992314b1f77bee7e5dbd40628d263c5109009166fdfada946bc65109a Aug 04 16:10:20.589880 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:20.589835 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" event={"ID":"bdb0fed7-1554-4d6c-b5c3-312c8c8ef643","Type":"ContainerStarted","Data":"da01c74992314b1f77bee7e5dbd40628d263c5109009166fdfada946bc65109a"} Aug 04 16:10:23.605276 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:23.605238 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" event={"ID":"bdb0fed7-1554-4d6c-b5c3-312c8c8ef643","Type":"ContainerStarted","Data":"55798f6a2f5f4029b71e43c26663d62ec021ab9e912b109a674aba62c5d1f2f4"} Aug 04 16:10:23.605666 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:23.605434 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" Aug 04 16:10:23.658746 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:23.658701 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" podStartSLOduration=2.04215087 podStartE2EDuration="4.658686133s" totalImagesPullingTime="2.616535263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:10:23.657659111 +0000 UTC m=+524.801991372" watchObservedRunningTime="2026-08-04 16:10:23.658686133 +0000 UTC m=+524.803018379" Aug 04 16:10:34.611849 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:34.611775 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" Aug 04 16:10:47.471296 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.471254 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc"] Aug 04 16:10:47.471717 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.471508 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" podUID="bdb0fed7-1554-4d6c-b5c3-312c8c8ef643" containerName="manager" containerID="cri-o://55798f6a2f5f4029b71e43c26663d62ec021ab9e912b109a674aba62c5d1f2f4" gracePeriod=2 Aug 04 16:10:47.477838 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.477813 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc"] Aug 04 16:10:47.502901 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.502877 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pbrhj"] Aug 04 16:10:47.503210 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.503197 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb0fed7-1554-4d6c-b5c3-312c8c8ef643" containerName="manager" Aug 04 16:10:47.503249 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.503211 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb0fed7-1554-4d6c-b5c3-312c8c8ef643" containerName="manager" Aug 04 16:10:47.503300 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.503275 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb0fed7-1554-4d6c-b5c3-312c8c8ef643" containerName="manager" Aug 04 16:10:47.506324 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.506306 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pbrhj" Aug 04 16:10:47.509216 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:10:47.509185 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zpslc\" is forbidden: User \"system:node:ip-10-0-132-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-42.ec2.internal' and this object" podUID="bdb0fed7-1554-4d6c-b5c3-312c8c8ef643" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" Aug 04 16:10:47.516541 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.516519 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pbrhj"] Aug 04 16:10:47.690377 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.690351 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfpf8\" (UniqueName: \"kubernetes.io/projected/a0d8a3ab-8c0c-4db0-8a2a-abfc30137224-kube-api-access-vfpf8\") pod \"limitador-operator-controller-manager-6c549b6d85-pbrhj\" (UID: \"a0d8a3ab-8c0c-4db0-8a2a-abfc30137224\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pbrhj" Aug 04 16:10:47.706510 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.706485 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" Aug 04 16:10:47.708023 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.708001 2580 generic.go:356] "Generic (PLEG): container finished" podID="bdb0fed7-1554-4d6c-b5c3-312c8c8ef643" containerID="55798f6a2f5f4029b71e43c26663d62ec021ab9e912b109a674aba62c5d1f2f4" exitCode=0 Aug 04 16:10:47.708115 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.708040 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" Aug 04 16:10:47.708115 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.708095 2580 scope.go:117] "RemoveContainer" containerID="55798f6a2f5f4029b71e43c26663d62ec021ab9e912b109a674aba62c5d1f2f4" Aug 04 16:10:47.709085 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:10:47.708951 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zpslc\" is forbidden: User \"system:node:ip-10-0-132-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-42.ec2.internal' and this object" podUID="bdb0fed7-1554-4d6c-b5c3-312c8c8ef643" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" Aug 04 16:10:47.714511 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:10:47.714486 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zpslc\" is forbidden: User \"system:node:ip-10-0-132-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-42.ec2.internal' and this object" podUID="bdb0fed7-1554-4d6c-b5c3-312c8c8ef643" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" Aug 04 16:10:47.715742 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.715725 2580 scope.go:117] "RemoveContainer" containerID="55798f6a2f5f4029b71e43c26663d62ec021ab9e912b109a674aba62c5d1f2f4" Aug 04 16:10:47.715994 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:10:47.715975 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55798f6a2f5f4029b71e43c26663d62ec021ab9e912b109a674aba62c5d1f2f4\": container with ID starting with 55798f6a2f5f4029b71e43c26663d62ec021ab9e912b109a674aba62c5d1f2f4 not found: ID does not exist" containerID="55798f6a2f5f4029b71e43c26663d62ec021ab9e912b109a674aba62c5d1f2f4" Aug 04 16:10:47.716063 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.716000 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55798f6a2f5f4029b71e43c26663d62ec021ab9e912b109a674aba62c5d1f2f4"} err="failed to get container status \"55798f6a2f5f4029b71e43c26663d62ec021ab9e912b109a674aba62c5d1f2f4\": rpc error: code = NotFound desc = could not find container \"55798f6a2f5f4029b71e43c26663d62ec021ab9e912b109a674aba62c5d1f2f4\": container with ID starting with 55798f6a2f5f4029b71e43c26663d62ec021ab9e912b109a674aba62c5d1f2f4 not found: ID does not exist" Aug 04 16:10:47.791252 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.791202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfpf8\" (UniqueName: \"kubernetes.io/projected/a0d8a3ab-8c0c-4db0-8a2a-abfc30137224-kube-api-access-vfpf8\") pod \"limitador-operator-controller-manager-6c549b6d85-pbrhj\" (UID: \"a0d8a3ab-8c0c-4db0-8a2a-abfc30137224\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pbrhj" Aug 04 16:10:47.800231 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.800214 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfpf8\" (UniqueName: \"kubernetes.io/projected/a0d8a3ab-8c0c-4db0-8a2a-abfc30137224-kube-api-access-vfpf8\") pod \"limitador-operator-controller-manager-6c549b6d85-pbrhj\" (UID: \"a0d8a3ab-8c0c-4db0-8a2a-abfc30137224\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pbrhj" Aug 04 16:10:47.853062 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.853040 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pbrhj" Aug 04 16:10:47.892553 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.892530 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtkkn\" (UniqueName: \"kubernetes.io/projected/bdb0fed7-1554-4d6c-b5c3-312c8c8ef643-kube-api-access-xtkkn\") pod \"bdb0fed7-1554-4d6c-b5c3-312c8c8ef643\" (UID: \"bdb0fed7-1554-4d6c-b5c3-312c8c8ef643\") " Aug 04 16:10:47.894546 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.894522 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb0fed7-1554-4d6c-b5c3-312c8c8ef643-kube-api-access-xtkkn" (OuterVolumeSpecName: "kube-api-access-xtkkn") pod "bdb0fed7-1554-4d6c-b5c3-312c8c8ef643" (UID: "bdb0fed7-1554-4d6c-b5c3-312c8c8ef643"). InnerVolumeSpecName "kube-api-access-xtkkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:10:47.979944 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.979922 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pbrhj"] Aug 04 16:10:47.982611 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:10:47.982587 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d8a3ab_8c0c_4db0_8a2a_abfc30137224.slice/crio-5e6d22b49f6974a3b79e37d1b9910d26ca1cde267f3afbbb787e4fcf35cfd879 WatchSource:0}: Error finding container 5e6d22b49f6974a3b79e37d1b9910d26ca1cde267f3afbbb787e4fcf35cfd879: Status 404 returned error can't find the container with id 5e6d22b49f6974a3b79e37d1b9910d26ca1cde267f3afbbb787e4fcf35cfd879 Aug 04 16:10:47.993432 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:47.993414 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtkkn\" (UniqueName: \"kubernetes.io/projected/bdb0fed7-1554-4d6c-b5c3-312c8c8ef643-kube-api-access-xtkkn\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:10:48.019345 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:10:48.019314 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zpslc\" is forbidden: User \"system:node:ip-10-0-132-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-42.ec2.internal' and this object" podUID="bdb0fed7-1554-4d6c-b5c3-312c8c8ef643" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" Aug 04 16:10:48.714244 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:48.714208 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pbrhj" event={"ID":"a0d8a3ab-8c0c-4db0-8a2a-abfc30137224","Type":"ContainerStarted","Data":"dd522c3049e98ca976c8c5b30b2362b44937ba35f269384f688fe590128d4973"} Aug 04 16:10:48.714641 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:48.714264 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pbrhj" event={"ID":"a0d8a3ab-8c0c-4db0-8a2a-abfc30137224","Type":"ContainerStarted","Data":"5e6d22b49f6974a3b79e37d1b9910d26ca1cde267f3afbbb787e4fcf35cfd879"} Aug 04 16:10:48.714641 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:48.714305 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pbrhj" Aug 04 16:10:48.735537 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:10:48.735501 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zpslc\" is forbidden: User \"system:node:ip-10-0-132-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-42.ec2.internal' and this object" podUID="bdb0fed7-1554-4d6c-b5c3-312c8c8ef643" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" Aug 04 16:10:48.735711 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:48.735678 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pbrhj" podStartSLOduration=1.73566377 podStartE2EDuration="1.73566377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:10:48.733387723 +0000 UTC m=+549.877719989" watchObservedRunningTime="2026-08-04 16:10:48.73566377 +0000 UTC m=+549.879996016" Aug 04 16:10:49.317815 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:10:49.317777 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-zpslc\" is forbidden: User \"system:node:ip-10-0-132-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-42.ec2.internal' and this object" podUID="bdb0fed7-1554-4d6c-b5c3-312c8c8ef643" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zpslc" Aug 04 16:10:49.319049 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:49.319026 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdb0fed7-1554-4d6c-b5c3-312c8c8ef643" path="/var/lib/kubelet/pods/bdb0fed7-1554-4d6c-b5c3-312c8c8ef643/volumes" Aug 04 16:10:59.721502 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:10:59.721470 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pbrhj" Aug 04 16:11:35.057277 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:35.057242 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:11:35.060712 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:35.060692 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb" Aug 04 16:11:35.063338 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:35.063316 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hhpjv\"" Aug 04 16:11:35.063575 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:35.063562 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 04 16:11:35.069845 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:35.069819 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:11:35.111046 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:35.111023 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:11:35.146969 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:35.146943 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l65nt\" (UniqueName: \"kubernetes.io/projected/aed77b61-81d6-40d8-b249-d61e28bbe868-kube-api-access-l65nt\") pod \"limitador-limitador-85d9db4d5d-rjxvb\" (UID: \"aed77b61-81d6-40d8-b249-d61e28bbe868\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb" Aug 04 16:11:35.147090 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:35.146971 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aed77b61-81d6-40d8-b249-d61e28bbe868-config-file\") pod \"limitador-limitador-85d9db4d5d-rjxvb\" (UID: \"aed77b61-81d6-40d8-b249-d61e28bbe868\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb" Aug 04 16:11:35.247609 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:35.247580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aed77b61-81d6-40d8-b249-d61e28bbe868-config-file\") pod \"limitador-limitador-85d9db4d5d-rjxvb\" (UID: \"aed77b61-81d6-40d8-b249-d61e28bbe868\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb" Aug 04 16:11:35.247770 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:35.247749 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l65nt\" (UniqueName: \"kubernetes.io/projected/aed77b61-81d6-40d8-b249-d61e28bbe868-kube-api-access-l65nt\") pod \"limitador-limitador-85d9db4d5d-rjxvb\" (UID: \"aed77b61-81d6-40d8-b249-d61e28bbe868\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb" Aug 04 16:11:35.248207 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:35.248186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aed77b61-81d6-40d8-b249-d61e28bbe868-config-file\") pod \"limitador-limitador-85d9db4d5d-rjxvb\" (UID: \"aed77b61-81d6-40d8-b249-d61e28bbe868\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb" Aug 04 16:11:35.256480 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:35.256457 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l65nt\" (UniqueName: \"kubernetes.io/projected/aed77b61-81d6-40d8-b249-d61e28bbe868-kube-api-access-l65nt\") pod \"limitador-limitador-85d9db4d5d-rjxvb\" (UID: \"aed77b61-81d6-40d8-b249-d61e28bbe868\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb" Aug 04 16:11:35.371021 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:35.371003 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb" Aug 04 16:11:35.506826 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:35.506802 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:11:35.508843 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:11:35.508817 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed77b61_81d6_40d8_b249_d61e28bbe868.slice/crio-ace449266177d7b36eae302dcd2c43724ec9dc29756ad1ebc29804f93c480761 WatchSource:0}: Error finding container ace449266177d7b36eae302dcd2c43724ec9dc29756ad1ebc29804f93c480761: Status 404 returned error can't find the container with id ace449266177d7b36eae302dcd2c43724ec9dc29756ad1ebc29804f93c480761 Aug 04 16:11:35.918152 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:35.918118 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb" event={"ID":"aed77b61-81d6-40d8-b249-d61e28bbe868","Type":"ContainerStarted","Data":"ace449266177d7b36eae302dcd2c43724ec9dc29756ad1ebc29804f93c480761"} Aug 04 16:11:36.924426 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:36.924392 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb" event={"ID":"aed77b61-81d6-40d8-b249-d61e28bbe868","Type":"ContainerStarted","Data":"46e4fa32c5bbe13fd4335ff8fc427d3fe17f358bfcc8b45d7229a1f63820765f"} Aug 04 16:11:36.924776 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:36.924527 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb" Aug 04 16:11:36.944048 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:36.944004 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb" podStartSLOduration=0.950454894 podStartE2EDuration="1.943991231s" totalImagesPullingTime="993.536337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:11:36.943370352 +0000 UTC m=+598.087702619" watchObservedRunningTime="2026-08-04 16:11:36.943991231 +0000 UTC m=+598.088323476" Aug 04 16:11:39.213705 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:39.213677 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:11:39.215602 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:39.215577 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:11:47.929495 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:11:47.929464 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb" Aug 04 16:12:50.129626 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:12:50.129593 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:13:07.839784 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:07.839754 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:13:13.633808 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:13.633768 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:13:22.140017 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:22.139982 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:13:31.816885 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:31.816797 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf"] Aug 04 16:13:31.820775 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:31.820753 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:31.824514 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:31.824493 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l4q6r\"" Aug 04 16:13:31.824514 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:31.824504 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Aug 04 16:13:31.824650 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:31.824498 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 16:13:31.824650 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:31.824577 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 16:13:31.831813 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:31.831793 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf"] Aug 04 16:13:31.971903 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:31.971876 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be9906f6-2bce-4957-a0cd-2311ab13d0f7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:31.972052 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:31.971913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbbc5\" (UniqueName: \"kubernetes.io/projected/be9906f6-2bce-4957-a0cd-2311ab13d0f7-kube-api-access-hbbc5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:31.972052 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:31.971933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be9906f6-2bce-4957-a0cd-2311ab13d0f7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:31.972052 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:31.971954 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be9906f6-2bce-4957-a0cd-2311ab13d0f7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:31.972052 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:31.971970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be9906f6-2bce-4957-a0cd-2311ab13d0f7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:31.972052 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:31.971996 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be9906f6-2bce-4957-a0cd-2311ab13d0f7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:32.072760 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:32.072740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbbc5\" (UniqueName: \"kubernetes.io/projected/be9906f6-2bce-4957-a0cd-2311ab13d0f7-kube-api-access-hbbc5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:32.072890 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:32.072770 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be9906f6-2bce-4957-a0cd-2311ab13d0f7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:32.072890 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:32.072792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be9906f6-2bce-4957-a0cd-2311ab13d0f7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:32.072890 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:32.072810 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be9906f6-2bce-4957-a0cd-2311ab13d0f7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:32.072890 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:32.072837 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be9906f6-2bce-4957-a0cd-2311ab13d0f7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:32.073081 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:32.072932 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be9906f6-2bce-4957-a0cd-2311ab13d0f7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:32.073229 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:32.073207 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be9906f6-2bce-4957-a0cd-2311ab13d0f7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:32.073319 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:32.073247 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be9906f6-2bce-4957-a0cd-2311ab13d0f7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:32.073319 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:32.073261 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be9906f6-2bce-4957-a0cd-2311ab13d0f7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:32.075056 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:32.075036 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be9906f6-2bce-4957-a0cd-2311ab13d0f7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:32.075314 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:32.075297 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be9906f6-2bce-4957-a0cd-2311ab13d0f7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:32.081075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:32.081056 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbbc5\" (UniqueName: \"kubernetes.io/projected/be9906f6-2bce-4957-a0cd-2311ab13d0f7-kube-api-access-hbbc5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf\" (UID: \"be9906f6-2bce-4957-a0cd-2311ab13d0f7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:32.130889 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:32.130868 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:32.275223 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:32.275191 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf"] Aug 04 16:13:32.278138 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:13:32.278111 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9906f6_2bce_4957_a0cd_2311ab13d0f7.slice/crio-020619ee67bc16e49f45bdb14024a3a36dee656d956b408c91e3668a16126647 WatchSource:0}: Error finding container 020619ee67bc16e49f45bdb14024a3a36dee656d956b408c91e3668a16126647: Status 404 returned error can't find the container with id 020619ee67bc16e49f45bdb14024a3a36dee656d956b408c91e3668a16126647 Aug 04 16:13:32.434254 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:32.434178 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" event={"ID":"be9906f6-2bce-4957-a0cd-2311ab13d0f7","Type":"ContainerStarted","Data":"020619ee67bc16e49f45bdb14024a3a36dee656d956b408c91e3668a16126647"} Aug 04 16:13:33.068650 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:33.068610 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:13:34.445953 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:34.445918 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" event={"ID":"be9906f6-2bce-4957-a0cd-2311ab13d0f7","Type":"ContainerStarted","Data":"0d53b62e32fc53906e818ed5772d2bdbf959035aa4335b984ad0fd0e9735a84d"} Aug 04 16:13:34.470392 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:34.470348 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" podStartSLOduration=2.081989273 podStartE2EDuration="3.470330626s" totalImagesPullingTime="1.388341353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:13:34.469020373 +0000 UTC m=+715.613352638" watchObservedRunningTime="2026-08-04 16:13:34.470330626 +0000 UTC m=+715.614662850" Aug 04 16:13:40.142329 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:40.142296 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:13:42.131795 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:42.131768 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:42.132159 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:42.131822 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:42.143772 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:42.143748 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:42.491791 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:42.491724 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf" Aug 04 16:13:47.602481 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.602444 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb"] Aug 04 16:13:47.605847 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.605829 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.608650 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.608630 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Aug 04 16:13:47.621364 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.621341 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb"] Aug 04 16:13:47.794428 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.794405 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b776ed5e-e090-40cd-9ccb-d97e80ff878e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.794540 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.794435 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b776ed5e-e090-40cd-9ccb-d97e80ff878e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.794598 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.794527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b776ed5e-e090-40cd-9ccb-d97e80ff878e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.794598 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.794566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b776ed5e-e090-40cd-9ccb-d97e80ff878e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.794680 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.794623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss6g8\" (UniqueName: \"kubernetes.io/projected/b776ed5e-e090-40cd-9ccb-d97e80ff878e-kube-api-access-ss6g8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.794680 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.794644 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b776ed5e-e090-40cd-9ccb-d97e80ff878e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.895877 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.895812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b776ed5e-e090-40cd-9ccb-d97e80ff878e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.895877 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.895844 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b776ed5e-e090-40cd-9ccb-d97e80ff878e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.896038 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.895890 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss6g8\" (UniqueName: \"kubernetes.io/projected/b776ed5e-e090-40cd-9ccb-d97e80ff878e-kube-api-access-ss6g8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.896038 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.895917 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b776ed5e-e090-40cd-9ccb-d97e80ff878e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.896038 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.895944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b776ed5e-e090-40cd-9ccb-d97e80ff878e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.896038 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.895965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b776ed5e-e090-40cd-9ccb-d97e80ff878e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.896372 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.896346 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b776ed5e-e090-40cd-9ccb-d97e80ff878e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.896477 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.896384 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b776ed5e-e090-40cd-9ccb-d97e80ff878e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.896477 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.896439 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b776ed5e-e090-40cd-9ccb-d97e80ff878e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.898070 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.898043 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b776ed5e-e090-40cd-9ccb-d97e80ff878e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.898456 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.898430 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b776ed5e-e090-40cd-9ccb-d97e80ff878e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.904982 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.904961 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss6g8\" (UniqueName: \"kubernetes.io/projected/b776ed5e-e090-40cd-9ccb-d97e80ff878e-kube-api-access-ss6g8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb\" (UID: \"b776ed5e-e090-40cd-9ccb-d97e80ff878e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:47.914697 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:47.914680 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:48.046162 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:48.046136 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb"] Aug 04 16:13:48.049155 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:13:48.049111 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb776ed5e_e090_40cd_9ccb_d97e80ff878e.slice/crio-343f3bddb64d96f2ef78829e70dd6b72e377e72978e7699ca4708b0d136a0fa2 WatchSource:0}: Error finding container 343f3bddb64d96f2ef78829e70dd6b72e377e72978e7699ca4708b0d136a0fa2: Status 404 returned error can't find the container with id 343f3bddb64d96f2ef78829e70dd6b72e377e72978e7699ca4708b0d136a0fa2 Aug 04 16:13:48.050678 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:48.050658 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:13:48.509110 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:48.509076 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" event={"ID":"b776ed5e-e090-40cd-9ccb-d97e80ff878e","Type":"ContainerStarted","Data":"45ebac0ea4219a3d9877e80d4d295b67984be44d5ca96af059c027e4fdf260e0"} Aug 04 16:13:48.509110 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:48.509113 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" event={"ID":"b776ed5e-e090-40cd-9ccb-d97e80ff878e","Type":"ContainerStarted","Data":"343f3bddb64d96f2ef78829e70dd6b72e377e72978e7699ca4708b0d136a0fa2"} Aug 04 16:13:48.530943 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:48.530900 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" podStartSLOduration=1.266432572 podStartE2EDuration="1.530883829s" totalImagesPullingTime="264.451257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:13:48.530670414 +0000 UTC m=+729.675002671" watchObservedRunningTime="2026-08-04 16:13:48.530883829 +0000 UTC m=+729.675216074" Aug 04 16:13:50.139863 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:50.139827 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:13:57.915472 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:57.915443 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:57.915472 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:57.915480 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:57.927169 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:57.927149 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:13:58.563010 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:13:58.562982 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb" Aug 04 16:14:49.347319 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:14:49.347270 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:14:59.151803 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:14:59.151770 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:15:11.944102 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:15:11.944025 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:16:39.245263 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:16:39.245195 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:16:39.245263 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:16:39.245195 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:16:53.442182 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:16:53.442150 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:17:02.946821 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:17:02.946787 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:18:01.442911 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:18:01.442835 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:18:08.837817 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:18:08.837781 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:18:25.231966 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:18:25.231921 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:18:31.140261 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:18:31.140215 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:18:40.634047 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:18:40.634014 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:18:47.727702 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:18:47.727670 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:19:01.335379 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:19:01.335346 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:19:06.236036 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:19:06.235996 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:19:14.840958 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:19:14.840923 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:19:20.543431 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:19:20.543399 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:19:33.336166 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:19:33.336091 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:19:44.234258 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:19:44.234221 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:20:07.963403 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:20:07.963371 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:20:18.540955 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:20:18.540923 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:20:24.145143 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:20:24.145111 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:20:32.844532 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:20:32.844499 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:20:43.746453 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:20:43.746412 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:20:50.058968 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:20:50.058932 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:20:57.232206 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:20:57.232172 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:21:11.539568 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:21:11.539494 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:21:16.541984 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:21:16.541953 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:21:27.638644 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:21:27.638616 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:21:32.632571 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:21:32.632530 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:21:38.844751 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:21:38.844721 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:21:39.271530 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:21:39.271450 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:21:39.274141 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:21:39.274115 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:21:47.830169 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:21:47.830133 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:21:53.529014 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:21:53.528983 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:22:04.634015 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:22:04.633983 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:24:21.730106 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:24:21.730032 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:24:26.636099 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:24:26.636062 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:24:50.843185 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:24:50.843143 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:25:16.444176 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:25:16.444143 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:25:22.835096 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:25:22.835060 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:25:32.538240 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:25:32.538166 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:25:38.339611 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:25:38.339571 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:25:47.939991 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:25:47.939955 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:25:54.640520 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:25:54.640473 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:26:03.251072 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:26:03.251037 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:26:09.897075 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:26:09.897041 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:26:18.831623 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:26:18.831589 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:26:25.634805 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:26:25.634771 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:26:34.930671 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:26:34.930639 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:26:39.300032 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:26:39.299994 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:26:39.303684 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:26:39.303661 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:26:57.135260 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:26:57.135222 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:27:03.837949 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:03.837867 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:27:16.133782 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:16.133751 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:27:31.985142 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:31.985108 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz"] Aug 04 16:27:31.989197 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:31.989173 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:31.991937 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:31.991913 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-186e23db-kserve-self-signed-certs\"" Aug 04 16:27:31.999929 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:31.999908 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz"] Aug 04 16:27:32.154723 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.154693 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bffd9e-6460-4700-876b-27c262f62958-tls-certs\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.154723 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.154724 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-home\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.154913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.154750 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-tmp-dir\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.154913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.154765 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff6tp\" (UniqueName: \"kubernetes.io/projected/07bffd9e-6460-4700-876b-27c262f62958-kube-api-access-ff6tp\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.154913 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.154885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-model-cache\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.155009 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.154918 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-dshm\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.255336 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.255247 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-model-cache\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.255336 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.255297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-dshm\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.255336 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.255323 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bffd9e-6460-4700-876b-27c262f62958-tls-certs\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.255559 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.255339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-home\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.255559 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.255362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-tmp-dir\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.255559 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.255379 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff6tp\" (UniqueName: \"kubernetes.io/projected/07bffd9e-6460-4700-876b-27c262f62958-kube-api-access-ff6tp\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.255748 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.255726 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-model-cache\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.255802 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.255786 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-tmp-dir\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.255838 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.255802 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-home\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.257517 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.257496 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-dshm\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.257731 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.257713 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bffd9e-6460-4700-876b-27c262f62958-tls-certs\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.264744 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.264722 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff6tp\" (UniqueName: \"kubernetes.io/projected/07bffd9e-6460-4700-876b-27c262f62958-kube-api-access-ff6tp\") pod \"e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.298644 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.298616 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:32.437814 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:32.437790 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz"] Aug 04 16:27:32.440074 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:27:32.440049 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07bffd9e_6460_4700_876b_27c262f62958.slice/crio-bee5ff71310fc2a9113c43f5c259ce1cf624975bc5fce8e0f664df7829152c17 WatchSource:0}: Error finding container bee5ff71310fc2a9113c43f5c259ce1cf624975bc5fce8e0f664df7829152c17: Status 404 returned error can't find the container with id bee5ff71310fc2a9113c43f5c259ce1cf624975bc5fce8e0f664df7829152c17 Aug 04 16:27:33.151152 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:33.151119 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" event={"ID":"07bffd9e-6460-4700-876b-27c262f62958","Type":"ContainerStarted","Data":"b561955adcc77c05123f6b60a1c60f1b083b23ebd26cc6f294c1d0a155cdf994"} Aug 04 16:27:33.151538 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:33.151160 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" event={"ID":"07bffd9e-6460-4700-876b-27c262f62958","Type":"ContainerStarted","Data":"bee5ff71310fc2a9113c43f5c259ce1cf624975bc5fce8e0f664df7829152c17"} Aug 04 16:27:33.168756 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:33.168705 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" podStartSLOduration=2.16868764 podStartE2EDuration="2.16868764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:27:33.168007039 +0000 UTC m=+1554.312339284" watchObservedRunningTime="2026-08-04 16:27:33.16868764 +0000 UTC m=+1554.313019890" Aug 04 16:27:34.446304 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:34.446256 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:27:42.299020 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:42.298986 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:42.299405 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:42.299041 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:42.311019 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:42.310990 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:27:43.206109 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:27:43.206079 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:28:03.650964 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:03.650932 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz"] Aug 04 16:28:03.839293 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:03.839251 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:28:14.548827 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:14.548794 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:28:18.669259 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:18.669213 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" podUID="07bffd9e-6460-4700-876b-27c262f62958" containerName="main" containerID="cri-o://b561955adcc77c05123f6b60a1c60f1b083b23ebd26cc6f294c1d0a155cdf994" gracePeriod=45 Aug 04 16:28:18.916378 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:18.916357 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:28:19.020385 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.020317 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-dshm\") pod \"07bffd9e-6460-4700-876b-27c262f62958\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " Aug 04 16:28:19.020385 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.020373 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff6tp\" (UniqueName: \"kubernetes.io/projected/07bffd9e-6460-4700-876b-27c262f62958-kube-api-access-ff6tp\") pod \"07bffd9e-6460-4700-876b-27c262f62958\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " Aug 04 16:28:19.020564 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.020406 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bffd9e-6460-4700-876b-27c262f62958-tls-certs\") pod \"07bffd9e-6460-4700-876b-27c262f62958\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " Aug 04 16:28:19.020564 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.020454 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-model-cache\") pod \"07bffd9e-6460-4700-876b-27c262f62958\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " Aug 04 16:28:19.020564 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.020504 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-tmp-dir\") pod \"07bffd9e-6460-4700-876b-27c262f62958\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " Aug 04 16:28:19.020564 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.020534 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-home\") pod \"07bffd9e-6460-4700-876b-27c262f62958\" (UID: \"07bffd9e-6460-4700-876b-27c262f62958\") " Aug 04 16:28:19.020807 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.020758 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-model-cache" (OuterVolumeSpecName: "model-cache") pod "07bffd9e-6460-4700-876b-27c262f62958" (UID: "07bffd9e-6460-4700-876b-27c262f62958"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:28:19.020854 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.020767 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07bffd9e-6460-4700-876b-27c262f62958" (UID: "07bffd9e-6460-4700-876b-27c262f62958"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:28:19.020886 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.020840 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-home" (OuterVolumeSpecName: "home") pod "07bffd9e-6460-4700-876b-27c262f62958" (UID: "07bffd9e-6460-4700-876b-27c262f62958"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:28:19.022557 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.022530 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bffd9e-6460-4700-876b-27c262f62958-kube-api-access-ff6tp" (OuterVolumeSpecName: "kube-api-access-ff6tp") pod "07bffd9e-6460-4700-876b-27c262f62958" (UID: "07bffd9e-6460-4700-876b-27c262f62958"). InnerVolumeSpecName "kube-api-access-ff6tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:28:19.022883 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.022858 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07bffd9e-6460-4700-876b-27c262f62958-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07bffd9e-6460-4700-876b-27c262f62958" (UID: "07bffd9e-6460-4700-876b-27c262f62958"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:28:19.022883 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.022864 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-dshm" (OuterVolumeSpecName: "dshm") pod "07bffd9e-6460-4700-876b-27c262f62958" (UID: "07bffd9e-6460-4700-876b-27c262f62958"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:28:19.121150 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.121119 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-home\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:28:19.121150 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.121146 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-dshm\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:28:19.121150 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.121155 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff6tp\" (UniqueName: \"kubernetes.io/projected/07bffd9e-6460-4700-876b-27c262f62958-kube-api-access-ff6tp\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:28:19.121326 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.121165 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bffd9e-6460-4700-876b-27c262f62958-tls-certs\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:28:19.121326 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.121174 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-model-cache\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:28:19.121326 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.121182 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bffd9e-6460-4700-876b-27c262f62958-tmp-dir\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:28:19.360270 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.360243 2580 generic.go:356] "Generic (PLEG): container finished" podID="07bffd9e-6460-4700-876b-27c262f62958" containerID="b561955adcc77c05123f6b60a1c60f1b083b23ebd26cc6f294c1d0a155cdf994" exitCode=0 Aug 04 16:28:19.360373 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.360302 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" Aug 04 16:28:19.360373 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.360311 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" event={"ID":"07bffd9e-6460-4700-876b-27c262f62958","Type":"ContainerDied","Data":"b561955adcc77c05123f6b60a1c60f1b083b23ebd26cc6f294c1d0a155cdf994"} Aug 04 16:28:19.360373 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.360353 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz" event={"ID":"07bffd9e-6460-4700-876b-27c262f62958","Type":"ContainerDied","Data":"bee5ff71310fc2a9113c43f5c259ce1cf624975bc5fce8e0f664df7829152c17"} Aug 04 16:28:19.360373 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.360371 2580 scope.go:117] "RemoveContainer" containerID="b561955adcc77c05123f6b60a1c60f1b083b23ebd26cc6f294c1d0a155cdf994" Aug 04 16:28:19.368562 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.368544 2580 scope.go:117] "RemoveContainer" containerID="b561955adcc77c05123f6b60a1c60f1b083b23ebd26cc6f294c1d0a155cdf994" Aug 04 16:28:19.368812 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:28:19.368793 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b561955adcc77c05123f6b60a1c60f1b083b23ebd26cc6f294c1d0a155cdf994\": container with ID starting with b561955adcc77c05123f6b60a1c60f1b083b23ebd26cc6f294c1d0a155cdf994 not found: ID does not exist" containerID="b561955adcc77c05123f6b60a1c60f1b083b23ebd26cc6f294c1d0a155cdf994" Aug 04 16:28:19.368862 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.368819 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b561955adcc77c05123f6b60a1c60f1b083b23ebd26cc6f294c1d0a155cdf994"} err="failed to get container status \"b561955adcc77c05123f6b60a1c60f1b083b23ebd26cc6f294c1d0a155cdf994\": rpc error: code = NotFound desc = could not find container \"b561955adcc77c05123f6b60a1c60f1b083b23ebd26cc6f294c1d0a155cdf994\": container with ID starting with b561955adcc77c05123f6b60a1c60f1b083b23ebd26cc6f294c1d0a155cdf994 not found: ID does not exist" Aug 04 16:28:19.381620 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.381596 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz"] Aug 04 16:28:19.386527 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.386496 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-186e23db-kserve-5d7c857599-q8kfz"] Aug 04 16:28:19.439686 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:19.439662 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:28:21.318357 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:21.318326 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07bffd9e-6460-4700-876b-27c262f62958" path="/var/lib/kubelet/pods/07bffd9e-6460-4700-876b-27c262f62958/volumes" Aug 04 16:28:25.933759 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:25.933727 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:28:36.136235 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:36.136146 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:28:42.340029 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:42.339993 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:28:48.732925 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:48.732893 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:28:55.332267 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:55.332235 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:28:59.833775 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:28:59.833739 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:29:08.845832 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:29:08.845795 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:29:15.343078 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:29:15.343045 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:29:31.649181 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:29:31.649149 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:29:38.141074 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:29:38.141036 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:29:50.144836 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:29:50.144799 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:29:58.136973 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:29:58.136938 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:30:04.240439 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:30:04.240351 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:30:09.138672 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:30:09.138640 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:30:15.339409 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:30:15.339375 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:30:25.531291 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:30:25.531259 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:30:31.734641 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:30:31.734612 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:30:42.642394 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:30:42.642362 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:30:48.936980 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:30:48.936943 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:30:59.932707 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:30:59.932671 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:31:15.130977 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:31:15.130917 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:31:21.126467 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:31:21.126431 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:31:27.339357 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:31:27.339325 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:31:39.151234 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:31:39.151198 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:31:39.326702 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:31:39.326678 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:31:39.332508 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:31:39.332485 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:31:47.435716 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:31:47.435646 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:31:52.738130 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:31:52.738099 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:32:07.534358 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:07.534325 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:32:29.248003 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:29.247966 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d639331-lkg56"] Aug 04 16:32:29.248420 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:29.248363 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07bffd9e-6460-4700-876b-27c262f62958" containerName="main" Aug 04 16:32:29.248420 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:29.248375 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07bffd9e-6460-4700-876b-27c262f62958" containerName="main" Aug 04 16:32:29.248489 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:29.248436 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="07bffd9e-6460-4700-876b-27c262f62958" containerName="main" Aug 04 16:32:29.251453 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:29.251435 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d639331-lkg56" Aug 04 16:32:29.254223 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:29.254202 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:32:29.254334 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:29.254205 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4pxfk\"" Aug 04 16:32:29.258895 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:29.258754 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d639331-lkg56"] Aug 04 16:32:29.405318 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:29.405270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63942482-347f-446a-a667-28f4f6260ead-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d639331-lkg56\" (UID: \"63942482-347f-446a-a667-28f4f6260ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d639331-lkg56" Aug 04 16:32:29.506659 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:29.506594 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63942482-347f-446a-a667-28f4f6260ead-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d639331-lkg56\" (UID: \"63942482-347f-446a-a667-28f4f6260ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d639331-lkg56" Aug 04 16:32:29.507147 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:29.507127 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63942482-347f-446a-a667-28f4f6260ead-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d639331-lkg56\" (UID: \"63942482-347f-446a-a667-28f4f6260ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d639331-lkg56" Aug 04 16:32:29.562773 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:29.562750 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d639331-lkg56" Aug 04 16:32:29.691984 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:29.691957 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d639331-lkg56"] Aug 04 16:32:29.692835 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:32:29.692806 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63942482_347f_446a_a667_28f4f6260ead.slice/crio-5dd56442c5299f89d819695a271658aaf7e65f603598d22b069fbf1e6a926f2e WatchSource:0}: Error finding container 5dd56442c5299f89d819695a271658aaf7e65f603598d22b069fbf1e6a926f2e: Status 404 returned error can't find the container with id 5dd56442c5299f89d819695a271658aaf7e65f603598d22b069fbf1e6a926f2e Aug 04 16:32:29.694556 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:29.694539 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:32:30.464245 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:30.464210 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d639331-lkg56" event={"ID":"63942482-347f-446a-a667-28f4f6260ead","Type":"ContainerStarted","Data":"5dd56442c5299f89d819695a271658aaf7e65f603598d22b069fbf1e6a926f2e"} Aug 04 16:32:39.506987 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:39.506957 2580 generic.go:356] "Generic (PLEG): container finished" podID="63942482-347f-446a-a667-28f4f6260ead" containerID="e69d59d2691a1ed7dc1ab16706cde455065a4226422509642190b3bc649fa4a1" exitCode=0 Aug 04 16:32:39.507266 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:39.507036 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d639331-lkg56" event={"ID":"63942482-347f-446a-a667-28f4f6260ead","Type":"ContainerDied","Data":"e69d59d2691a1ed7dc1ab16706cde455065a4226422509642190b3bc649fa4a1"} Aug 04 16:32:40.642844 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:40.642820 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d639331-lkg56" Aug 04 16:32:40.801527 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:40.801456 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63942482-347f-446a-a667-28f4f6260ead-maas-api-service-ca\") pod \"63942482-347f-446a-a667-28f4f6260ead\" (UID: \"63942482-347f-446a-a667-28f4f6260ead\") " Aug 04 16:32:40.801801 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:40.801783 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63942482-347f-446a-a667-28f4f6260ead-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63942482-347f-446a-a667-28f4f6260ead" (UID: "63942482-347f-446a-a667-28f4f6260ead"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:32:40.902850 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:40.902817 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63942482-347f-446a-a667-28f4f6260ead-maas-api-service-ca\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:32:41.517293 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:41.517259 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d639331-lkg56" Aug 04 16:32:41.517293 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:41.517269 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d639331-lkg56" event={"ID":"63942482-347f-446a-a667-28f4f6260ead","Type":"ContainerDied","Data":"5dd56442c5299f89d819695a271658aaf7e65f603598d22b069fbf1e6a926f2e"} Aug 04 16:32:41.517456 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:41.517329 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dd56442c5299f89d819695a271658aaf7e65f603598d22b069fbf1e6a926f2e" Aug 04 16:32:45.506090 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:45.506057 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d639331-lkg56"] Aug 04 16:32:45.508403 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:45.508379 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d639331-lkg56"] Aug 04 16:32:47.319215 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:47.319182 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63942482-347f-446a-a667-28f4f6260ead" path="/var/lib/kubelet/pods/63942482-347f-446a-a667-28f4f6260ead/volumes" Aug 04 16:32:48.736767 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:48.736734 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:32:56.555415 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:32:56.555383 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:33:13.940754 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:13.940724 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4"] Aug 04 16:33:13.941110 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:13.941075 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63942482-347f-446a-a667-28f4f6260ead" containerName="revoke-keys" Aug 04 16:33:13.941110 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:13.941088 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="63942482-347f-446a-a667-28f4f6260ead" containerName="revoke-keys" Aug 04 16:33:13.941187 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:13.941149 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="63942482-347f-446a-a667-28f4f6260ead" containerName="revoke-keys" Aug 04 16:33:13.944176 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:13.944161 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4" Aug 04 16:33:13.947272 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:13.947253 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4pxfk\"" Aug 04 16:33:13.947402 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:13.947369 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:33:13.953099 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:13.953080 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4"] Aug 04 16:33:14.046495 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:14.046468 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f78c9f-1389-47bc-bc52-e26127a7f19a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4\" (UID: \"62f78c9f-1389-47bc-bc52-e26127a7f19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4" Aug 04 16:33:14.146742 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:14.146716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f78c9f-1389-47bc-bc52-e26127a7f19a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4\" (UID: \"62f78c9f-1389-47bc-bc52-e26127a7f19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4" Aug 04 16:33:14.147264 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:14.147247 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f78c9f-1389-47bc-bc52-e26127a7f19a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4\" (UID: \"62f78c9f-1389-47bc-bc52-e26127a7f19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4" Aug 04 16:33:14.255637 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:14.255579 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4" Aug 04 16:33:14.383149 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:14.383127 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4"] Aug 04 16:33:14.385722 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:33:14.385696 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f78c9f_1389_47bc_bc52_e26127a7f19a.slice/crio-355bdf38d32976208d6b4091166c74073a675bc4ceff5adfa59c8ecd1c748fa6 WatchSource:0}: Error finding container 355bdf38d32976208d6b4091166c74073a675bc4ceff5adfa59c8ecd1c748fa6: Status 404 returned error can't find the container with id 355bdf38d32976208d6b4091166c74073a675bc4ceff5adfa59c8ecd1c748fa6 Aug 04 16:33:14.667550 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:14.667520 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4" event={"ID":"62f78c9f-1389-47bc-bc52-e26127a7f19a","Type":"ContainerStarted","Data":"1d5e7ca09198ef1547add9d682af8475d158bb380cdffa363f57641835d7890f"} Aug 04 16:33:14.667725 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:14.667575 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4" event={"ID":"62f78c9f-1389-47bc-bc52-e26127a7f19a","Type":"ContainerStarted","Data":"355bdf38d32976208d6b4091166c74073a675bc4ceff5adfa59c8ecd1c748fa6"} Aug 04 16:33:14.683156 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:14.683114 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4" podStartSLOduration=1.683099303 podStartE2EDuration="1.683099303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:33:14.68196801 +0000 UTC m=+1895.826300275" watchObservedRunningTime="2026-08-04 16:33:14.683099303 +0000 UTC m=+1895.827431545" Aug 04 16:33:15.673460 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:15.673427 2580 generic.go:356] "Generic (PLEG): container finished" podID="62f78c9f-1389-47bc-bc52-e26127a7f19a" containerID="1d5e7ca09198ef1547add9d682af8475d158bb380cdffa363f57641835d7890f" exitCode=0 Aug 04 16:33:15.673879 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:15.673506 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4" event={"ID":"62f78c9f-1389-47bc-bc52-e26127a7f19a","Type":"ContainerDied","Data":"1d5e7ca09198ef1547add9d682af8475d158bb380cdffa363f57641835d7890f"} Aug 04 16:33:16.814053 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:16.814022 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4" Aug 04 16:33:16.872050 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:16.872019 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f78c9f-1389-47bc-bc52-e26127a7f19a-maas-api-service-ca\") pod \"62f78c9f-1389-47bc-bc52-e26127a7f19a\" (UID: \"62f78c9f-1389-47bc-bc52-e26127a7f19a\") " Aug 04 16:33:16.872381 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:16.872360 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f78c9f-1389-47bc-bc52-e26127a7f19a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62f78c9f-1389-47bc-bc52-e26127a7f19a" (UID: "62f78c9f-1389-47bc-bc52-e26127a7f19a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:33:16.973623 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:16.973547 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f78c9f-1389-47bc-bc52-e26127a7f19a-maas-api-service-ca\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:33:17.684613 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:17.684527 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4" Aug 04 16:33:17.684613 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:17.684529 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4" event={"ID":"62f78c9f-1389-47bc-bc52-e26127a7f19a","Type":"ContainerDied","Data":"355bdf38d32976208d6b4091166c74073a675bc4ceff5adfa59c8ecd1c748fa6"} Aug 04 16:33:17.684833 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:17.684650 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="355bdf38d32976208d6b4091166c74073a675bc4ceff5adfa59c8ecd1c748fa6" Aug 04 16:33:19.982021 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:19.981988 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4"] Aug 04 16:33:19.984914 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:19.984888 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57d47eb1-4wjg4"] Aug 04 16:33:21.318912 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:21.318885 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62f78c9f-1389-47bc-bc52-e26127a7f19a" path="/var/lib/kubelet/pods/62f78c9f-1389-47bc-bc52-e26127a7f19a/volumes" Aug 04 16:33:22.836251 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:22.836218 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:33:32.244881 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:32.244850 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:33:48.996596 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:48.996561 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9"] Aug 04 16:33:48.997051 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:48.996925 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f78c9f-1389-47bc-bc52-e26127a7f19a" containerName="revoke-keys" Aug 04 16:33:48.997051 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:48.996942 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f78c9f-1389-47bc-bc52-e26127a7f19a" containerName="revoke-keys" Aug 04 16:33:48.997051 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:48.997020 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f78c9f-1389-47bc-bc52-e26127a7f19a" containerName="revoke-keys" Aug 04 16:33:48.999959 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:48.999942 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9" Aug 04 16:33:49.003011 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:49.002992 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4pxfk\"" Aug 04 16:33:49.003117 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:49.002990 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:33:49.010081 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:49.010060 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9"] Aug 04 16:33:49.125049 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:49.125023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8aeeda-1b98-4550-bc3a-a3827159776c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-160e6699-2tzq9\" (UID: \"8e8aeeda-1b98-4550-bc3a-a3827159776c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9" Aug 04 16:33:49.225972 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:49.225948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8aeeda-1b98-4550-bc3a-a3827159776c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-160e6699-2tzq9\" (UID: \"8e8aeeda-1b98-4550-bc3a-a3827159776c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9" Aug 04 16:33:49.226505 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:49.226486 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8aeeda-1b98-4550-bc3a-a3827159776c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-160e6699-2tzq9\" (UID: \"8e8aeeda-1b98-4550-bc3a-a3827159776c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9" Aug 04 16:33:49.311093 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:49.311036 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9" Aug 04 16:33:49.439121 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:49.438974 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9"] Aug 04 16:33:49.441889 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:33:49.441864 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8aeeda_1b98_4550_bc3a_a3827159776c.slice/crio-491a1e9b6e4e0f16f095462a31b9e459181e546c31b3b5496e2f4c0450ef565d WatchSource:0}: Error finding container 491a1e9b6e4e0f16f095462a31b9e459181e546c31b3b5496e2f4c0450ef565d: Status 404 returned error can't find the container with id 491a1e9b6e4e0f16f095462a31b9e459181e546c31b3b5496e2f4c0450ef565d Aug 04 16:33:49.834757 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:49.834720 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9" event={"ID":"8e8aeeda-1b98-4550-bc3a-a3827159776c","Type":"ContainerStarted","Data":"e656cc6bab0554760c4262534a6a7c4c5918aba3a942ff7239918e9d976157c5"} Aug 04 16:33:49.834881 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:49.834777 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9" event={"ID":"8e8aeeda-1b98-4550-bc3a-a3827159776c","Type":"ContainerStarted","Data":"491a1e9b6e4e0f16f095462a31b9e459181e546c31b3b5496e2f4c0450ef565d"} Aug 04 16:33:49.851074 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:49.851011 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9" podStartSLOduration=1.850996521 podStartE2EDuration="1.850996521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:33:49.850778436 +0000 UTC m=+1930.995110700" watchObservedRunningTime="2026-08-04 16:33:49.850996521 +0000 UTC m=+1930.995328766" Aug 04 16:33:50.841108 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:50.841072 2580 generic.go:356] "Generic (PLEG): container finished" podID="8e8aeeda-1b98-4550-bc3a-a3827159776c" containerID="e656cc6bab0554760c4262534a6a7c4c5918aba3a942ff7239918e9d976157c5" exitCode=0 Aug 04 16:33:50.841453 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:50.841139 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9" event={"ID":"8e8aeeda-1b98-4550-bc3a-a3827159776c","Type":"ContainerDied","Data":"e656cc6bab0554760c4262534a6a7c4c5918aba3a942ff7239918e9d976157c5"} Aug 04 16:33:51.980551 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:51.980523 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9" Aug 04 16:33:52.148434 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:52.148370 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8aeeda-1b98-4550-bc3a-a3827159776c-maas-api-service-ca\") pod \"8e8aeeda-1b98-4550-bc3a-a3827159776c\" (UID: \"8e8aeeda-1b98-4550-bc3a-a3827159776c\") " Aug 04 16:33:52.148702 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:52.148677 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8aeeda-1b98-4550-bc3a-a3827159776c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e8aeeda-1b98-4550-bc3a-a3827159776c" (UID: "8e8aeeda-1b98-4550-bc3a-a3827159776c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:33:52.248874 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:52.248848 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8aeeda-1b98-4550-bc3a-a3827159776c-maas-api-service-ca\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:33:52.851389 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:52.851359 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9" Aug 04 16:33:52.851389 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:52.851371 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9" event={"ID":"8e8aeeda-1b98-4550-bc3a-a3827159776c","Type":"ContainerDied","Data":"491a1e9b6e4e0f16f095462a31b9e459181e546c31b3b5496e2f4c0450ef565d"} Aug 04 16:33:52.851582 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:52.851424 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="491a1e9b6e4e0f16f095462a31b9e459181e546c31b3b5496e2f4c0450ef565d" Aug 04 16:33:54.998500 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:54.998448 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9"] Aug 04 16:33:55.000079 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:55.000060 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-160e6699-2tzq9"] Aug 04 16:33:55.319130 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:55.319041 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e8aeeda-1b98-4550-bc3a-a3827159776c" path="/var/lib/kubelet/pods/8e8aeeda-1b98-4550-bc3a-a3827159776c/volumes" Aug 04 16:33:57.875144 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:33:57.875103 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:34:05.941877 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:05.941842 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:34:24.293801 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:24.293768 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75b82354-bw7vc"] Aug 04 16:34:24.294330 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:24.294142 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e8aeeda-1b98-4550-bc3a-a3827159776c" containerName="revoke-keys" Aug 04 16:34:24.294330 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:24.294160 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e8aeeda-1b98-4550-bc3a-a3827159776c" containerName="revoke-keys" Aug 04 16:34:24.294330 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:24.294231 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e8aeeda-1b98-4550-bc3a-a3827159776c" containerName="revoke-keys" Aug 04 16:34:24.297215 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:24.297199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75b82354-bw7vc" Aug 04 16:34:24.299950 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:24.299933 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4pxfk\"" Aug 04 16:34:24.300041 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:24.299994 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:34:24.304944 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:24.304917 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75b82354-bw7vc"] Aug 04 16:34:24.408827 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:24.408800 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0199beb2-29a5-41cd-b7d3-7d8b39d48f43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75b82354-bw7vc\" (UID: \"0199beb2-29a5-41cd-b7d3-7d8b39d48f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75b82354-bw7vc" Aug 04 16:34:24.509532 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:24.509506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0199beb2-29a5-41cd-b7d3-7d8b39d48f43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75b82354-bw7vc\" (UID: \"0199beb2-29a5-41cd-b7d3-7d8b39d48f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75b82354-bw7vc" Aug 04 16:34:24.510130 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:24.510105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0199beb2-29a5-41cd-b7d3-7d8b39d48f43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75b82354-bw7vc\" (UID: \"0199beb2-29a5-41cd-b7d3-7d8b39d48f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75b82354-bw7vc" Aug 04 16:34:24.608947 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:24.608920 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75b82354-bw7vc" Aug 04 16:34:24.735328 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:24.735292 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75b82354-bw7vc"] Aug 04 16:34:24.738614 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:34:24.738587 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0199beb2_29a5_41cd_b7d3_7d8b39d48f43.slice/crio-3323fc8b730f764921708d21072fd72ddc84be9d1aea9ebe02a6daa8a707939b WatchSource:0}: Error finding container 3323fc8b730f764921708d21072fd72ddc84be9d1aea9ebe02a6daa8a707939b: Status 404 returned error can't find the container with id 3323fc8b730f764921708d21072fd72ddc84be9d1aea9ebe02a6daa8a707939b Aug 04 16:34:25.013178 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:25.013144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75b82354-bw7vc" event={"ID":"0199beb2-29a5-41cd-b7d3-7d8b39d48f43","Type":"ContainerStarted","Data":"3323fc8b730f764921708d21072fd72ddc84be9d1aea9ebe02a6daa8a707939b"} Aug 04 16:34:26.024986 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:26.024951 2580 generic.go:356] "Generic (PLEG): container finished" podID="0199beb2-29a5-41cd-b7d3-7d8b39d48f43" containerID="4996a268902b36b7f9a7b6bb216c147fcca51a6cbdd76abc33164350b493d19e" exitCode=0 Aug 04 16:34:26.025377 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:26.025039 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75b82354-bw7vc" event={"ID":"0199beb2-29a5-41cd-b7d3-7d8b39d48f43","Type":"ContainerDied","Data":"4996a268902b36b7f9a7b6bb216c147fcca51a6cbdd76abc33164350b493d19e"} Aug 04 16:34:27.154066 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:27.154044 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75b82354-bw7vc" Aug 04 16:34:27.233548 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:27.233521 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0199beb2-29a5-41cd-b7d3-7d8b39d48f43-maas-api-service-ca\") pod \"0199beb2-29a5-41cd-b7d3-7d8b39d48f43\" (UID: \"0199beb2-29a5-41cd-b7d3-7d8b39d48f43\") " Aug 04 16:34:27.233860 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:27.233835 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0199beb2-29a5-41cd-b7d3-7d8b39d48f43-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0199beb2-29a5-41cd-b7d3-7d8b39d48f43" (UID: "0199beb2-29a5-41cd-b7d3-7d8b39d48f43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:34:27.334395 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:27.334374 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0199beb2-29a5-41cd-b7d3-7d8b39d48f43-maas-api-service-ca\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:34:28.034526 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:28.034499 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75b82354-bw7vc" Aug 04 16:34:28.034678 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:28.034497 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75b82354-bw7vc" event={"ID":"0199beb2-29a5-41cd-b7d3-7d8b39d48f43","Type":"ContainerDied","Data":"3323fc8b730f764921708d21072fd72ddc84be9d1aea9ebe02a6daa8a707939b"} Aug 04 16:34:28.034678 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:28.034614 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3323fc8b730f764921708d21072fd72ddc84be9d1aea9ebe02a6daa8a707939b" Aug 04 16:34:31.257867 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:31.257831 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75b82354-bw7vc"] Aug 04 16:34:31.261428 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:31.261391 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75b82354-bw7vc"] Aug 04 16:34:31.319415 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:31.319381 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0199beb2-29a5-41cd-b7d3-7d8b39d48f43" path="/var/lib/kubelet/pods/0199beb2-29a5-41cd-b7d3-7d8b39d48f43/volumes" Aug 04 16:34:35.537424 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:35.537388 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:34:45.937241 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:34:45.937204 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:36:17.639800 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:36:17.639764 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:36:24.244820 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:36:24.244783 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:36:37.679311 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:36:37.679266 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:36:39.354205 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:36:39.354173 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:36:39.361315 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:36:39.361295 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:37:06.416778 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:06.416743 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph"] Aug 04 16:37:06.417212 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:06.417121 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0199beb2-29a5-41cd-b7d3-7d8b39d48f43" containerName="revoke-keys" Aug 04 16:37:06.417212 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:06.417133 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0199beb2-29a5-41cd-b7d3-7d8b39d48f43" containerName="revoke-keys" Aug 04 16:37:06.417212 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:06.417200 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0199beb2-29a5-41cd-b7d3-7d8b39d48f43" containerName="revoke-keys" Aug 04 16:37:06.420161 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:06.420144 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph" Aug 04 16:37:06.423266 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:06.423233 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4pxfk\"" Aug 04 16:37:06.423394 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:06.423296 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:37:06.429035 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:06.429010 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph"] Aug 04 16:37:06.571551 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:06.571522 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d297ab-7c08-45be-9279-8435d140e959-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph\" (UID: \"56d297ab-7c08-45be-9279-8435d140e959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph" Aug 04 16:37:06.672865 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:06.672808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d297ab-7c08-45be-9279-8435d140e959-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph\" (UID: \"56d297ab-7c08-45be-9279-8435d140e959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph" Aug 04 16:37:06.673424 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:06.673404 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d297ab-7c08-45be-9279-8435d140e959-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph\" (UID: \"56d297ab-7c08-45be-9279-8435d140e959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph" Aug 04 16:37:06.730942 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:06.730920 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph" Aug 04 16:37:06.859759 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:06.859729 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph"] Aug 04 16:37:06.860975 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:37:06.860943 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d297ab_7c08_45be_9279_8435d140e959.slice/crio-f3ddaa5f5dfb5a7630be395fac32fdc8119574fccd7c3491524e658d7e6bfcf3 WatchSource:0}: Error finding container f3ddaa5f5dfb5a7630be395fac32fdc8119574fccd7c3491524e658d7e6bfcf3: Status 404 returned error can't find the container with id f3ddaa5f5dfb5a7630be395fac32fdc8119574fccd7c3491524e658d7e6bfcf3 Aug 04 16:37:07.779834 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:07.779762 2580 generic.go:356] "Generic (PLEG): container finished" podID="56d297ab-7c08-45be-9279-8435d140e959" containerID="203e99a28bfaead7b7e9b0e11c2b30b29f8a784f73d60a90d0815d84da37456c" exitCode=0 Aug 04 16:37:07.780168 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:07.779844 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph" event={"ID":"56d297ab-7c08-45be-9279-8435d140e959","Type":"ContainerDied","Data":"203e99a28bfaead7b7e9b0e11c2b30b29f8a784f73d60a90d0815d84da37456c"} Aug 04 16:37:07.780168 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:07.779908 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph" event={"ID":"56d297ab-7c08-45be-9279-8435d140e959","Type":"ContainerStarted","Data":"f3ddaa5f5dfb5a7630be395fac32fdc8119574fccd7c3491524e658d7e6bfcf3"} Aug 04 16:37:08.914170 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:08.914148 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph" Aug 04 16:37:08.990818 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:08.990792 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d297ab-7c08-45be-9279-8435d140e959-maas-api-service-ca\") pod \"56d297ab-7c08-45be-9279-8435d140e959\" (UID: \"56d297ab-7c08-45be-9279-8435d140e959\") " Aug 04 16:37:08.991107 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:08.991087 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d297ab-7c08-45be-9279-8435d140e959-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "56d297ab-7c08-45be-9279-8435d140e959" (UID: "56d297ab-7c08-45be-9279-8435d140e959"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:37:09.091809 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:09.091789 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d297ab-7c08-45be-9279-8435d140e959-maas-api-service-ca\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:37:09.789465 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:09.789438 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph" Aug 04 16:37:09.789610 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:09.789479 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph" event={"ID":"56d297ab-7c08-45be-9279-8435d140e959","Type":"ContainerDied","Data":"f3ddaa5f5dfb5a7630be395fac32fdc8119574fccd7c3491524e658d7e6bfcf3"} Aug 04 16:37:09.789610 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:09.789530 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3ddaa5f5dfb5a7630be395fac32fdc8119574fccd7c3491524e658d7e6bfcf3" Aug 04 16:37:17.202127 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:17.202091 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph"] Aug 04 16:37:17.203971 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:17.203948 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f3e00e1-9jzph"] Aug 04 16:37:17.319422 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:17.319396 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56d297ab-7c08-45be-9279-8435d140e959" path="/var/lib/kubelet/pods/56d297ab-7c08-45be-9279-8435d140e959/volumes" Aug 04 16:37:18.842796 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:18.842765 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:37:56.742600 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:37:56.742565 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:38:39.423867 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:38:39.423841 2580 scope.go:117] "RemoveContainer" containerID="e69d59d2691a1ed7dc1ab16706cde455065a4226422509642190b3bc649fa4a1" Aug 04 16:39:39.442713 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:39:39.442683 2580 scope.go:117] "RemoveContainer" containerID="1d5e7ca09198ef1547add9d682af8475d158bb380cdffa363f57641835d7890f" Aug 04 16:39:43.534713 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:39:43.534683 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:39:51.344621 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:39:51.344590 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:40:18.143368 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:18.143329 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:40:34.434908 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:34.434831 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:40:39.460543 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:39.460514 2580 scope.go:117] "RemoveContainer" containerID="4996a268902b36b7f9a7b6bb216c147fcca51a6cbdd76abc33164350b493d19e" Aug 04 16:40:39.468581 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:39.468561 2580 scope.go:117] "RemoveContainer" containerID="e656cc6bab0554760c4262534a6a7c4c5918aba3a942ff7239918e9d976157c5" Aug 04 16:40:43.701363 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:43.701326 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t"] Aug 04 16:40:43.701881 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:43.701832 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d297ab-7c08-45be-9279-8435d140e959" containerName="revoke-keys" Aug 04 16:40:43.701881 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:43.701852 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d297ab-7c08-45be-9279-8435d140e959" containerName="revoke-keys" Aug 04 16:40:43.701997 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:43.701939 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="56d297ab-7c08-45be-9279-8435d140e959" containerName="revoke-keys" Aug 04 16:40:43.706449 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:43.706428 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t" Aug 04 16:40:43.709421 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:43.709400 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:40:43.709532 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:43.709431 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4pxfk\"" Aug 04 16:40:43.712883 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:43.712864 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t"] Aug 04 16:40:43.784883 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:43.784855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa6a858-c62e-49b0-abad-dd745b6b81d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8101352f-2kd2t\" (UID: \"9aa6a858-c62e-49b0-abad-dd745b6b81d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t" Aug 04 16:40:43.892417 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:43.892386 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa6a858-c62e-49b0-abad-dd745b6b81d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8101352f-2kd2t\" (UID: \"9aa6a858-c62e-49b0-abad-dd745b6b81d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t" Aug 04 16:40:43.893109 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:43.893090 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa6a858-c62e-49b0-abad-dd745b6b81d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8101352f-2kd2t\" (UID: \"9aa6a858-c62e-49b0-abad-dd745b6b81d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t" Aug 04 16:40:44.018036 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:44.017971 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t" Aug 04 16:40:44.141252 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:44.141214 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t"] Aug 04 16:40:44.144873 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:40:44.144845 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa6a858_c62e_49b0_abad_dd745b6b81d4.slice/crio-d811390488af8b040adeef5a6f4a478346e979e193029581737d8ca74e885fab WatchSource:0}: Error finding container d811390488af8b040adeef5a6f4a478346e979e193029581737d8ca74e885fab: Status 404 returned error can't find the container with id d811390488af8b040adeef5a6f4a478346e979e193029581737d8ca74e885fab Aug 04 16:40:44.777109 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:44.777027 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t" event={"ID":"9aa6a858-c62e-49b0-abad-dd745b6b81d4","Type":"ContainerStarted","Data":"bca19fbe3bfd766fc50881d5b75aca2ed51d2e71351f5c739569ec77b3f5125b"} Aug 04 16:40:44.777109 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:44.777080 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t" event={"ID":"9aa6a858-c62e-49b0-abad-dd745b6b81d4","Type":"ContainerStarted","Data":"d811390488af8b040adeef5a6f4a478346e979e193029581737d8ca74e885fab"} Aug 04 16:40:44.794460 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:44.794415 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t" podStartSLOduration=1.794399652 podStartE2EDuration="1.794399652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:40:44.793055389 +0000 UTC m=+2345.937387656" watchObservedRunningTime="2026-08-04 16:40:44.794399652 +0000 UTC m=+2345.938731897" Aug 04 16:40:45.783072 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:45.783036 2580 generic.go:356] "Generic (PLEG): container finished" podID="9aa6a858-c62e-49b0-abad-dd745b6b81d4" containerID="bca19fbe3bfd766fc50881d5b75aca2ed51d2e71351f5c739569ec77b3f5125b" exitCode=0 Aug 04 16:40:45.783458 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:45.783133 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t" event={"ID":"9aa6a858-c62e-49b0-abad-dd745b6b81d4","Type":"ContainerDied","Data":"bca19fbe3bfd766fc50881d5b75aca2ed51d2e71351f5c739569ec77b3f5125b"} Aug 04 16:40:46.915418 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:46.915394 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t" Aug 04 16:40:47.019959 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:47.019935 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa6a858-c62e-49b0-abad-dd745b6b81d4-maas-api-service-ca\") pod \"9aa6a858-c62e-49b0-abad-dd745b6b81d4\" (UID: \"9aa6a858-c62e-49b0-abad-dd745b6b81d4\") " Aug 04 16:40:47.020273 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:47.020251 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa6a858-c62e-49b0-abad-dd745b6b81d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9aa6a858-c62e-49b0-abad-dd745b6b81d4" (UID: "9aa6a858-c62e-49b0-abad-dd745b6b81d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:40:47.120956 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:47.120932 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa6a858-c62e-49b0-abad-dd745b6b81d4-maas-api-service-ca\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:40:47.792439 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:47.792413 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t" Aug 04 16:40:47.792565 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:47.792408 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t" event={"ID":"9aa6a858-c62e-49b0-abad-dd745b6b81d4","Type":"ContainerDied","Data":"d811390488af8b040adeef5a6f4a478346e979e193029581737d8ca74e885fab"} Aug 04 16:40:47.792565 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:47.792529 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d811390488af8b040adeef5a6f4a478346e979e193029581737d8ca74e885fab" Aug 04 16:40:56.850761 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:56.850721 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:40:59.440028 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:59.439994 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t"] Aug 04 16:40:59.442604 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:40:59.442580 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8101352f-2kd2t"] Aug 04 16:41:01.319388 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:41:01.319353 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aa6a858-c62e-49b0-abad-dd745b6b81d4" path="/var/lib/kubelet/pods/9aa6a858-c62e-49b0-abad-dd745b6b81d4/volumes" Aug 04 16:41:02.953739 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:41:02.953709 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:41:39.383147 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:41:39.383118 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:41:39.388958 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:41:39.388936 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:42:07.834932 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:42:07.834853 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:42:20.032684 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:42:20.032647 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:42:26.746704 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:42:26.746671 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:42:36.339023 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:42:36.338984 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:43:19.240047 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:43:19.240011 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:43:39.502218 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:43:39.502115 2580 scope.go:117] "RemoveContainer" containerID="203e99a28bfaead7b7e9b0e11c2b30b29f8a784f73d60a90d0815d84da37456c" Aug 04 16:43:49.434492 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:43:49.434457 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:44:28.116122 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.116079 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss"] Aug 04 16:44:28.116706 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.116634 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa6a858-c62e-49b0-abad-dd745b6b81d4" containerName="revoke-keys" Aug 04 16:44:28.116706 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.116656 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa6a858-c62e-49b0-abad-dd745b6b81d4" containerName="revoke-keys" Aug 04 16:44:28.116821 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.116765 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa6a858-c62e-49b0-abad-dd745b6b81d4" containerName="revoke-keys" Aug 04 16:44:28.119766 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.119744 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.123100 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.123075 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6c210e\"/\"openshift-service-ca.crt\"" Aug 04 16:44:28.124650 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.124627 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6c210e\"/\"default-dockercfg-9ftvk\"" Aug 04 16:44:28.124650 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.124643 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6c210e\"/\"kube-root-ca.crt\"" Aug 04 16:44:28.124815 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.124709 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6c210e\"/\"test-model-6c210e-kserve-self-signed-certs\"" Aug 04 16:44:28.136720 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.136697 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss"] Aug 04 16:44:28.193658 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.193631 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-tmp-dir\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.193789 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.193671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-home\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.193789 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.193740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-model-cache\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.193789 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.193777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lgjv\" (UniqueName: \"kubernetes.io/projected/9b2856ae-9b61-40c6-9738-de7425f39456-kube-api-access-8lgjv\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.193959 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.193805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b2856ae-9b61-40c6-9738-de7425f39456-tls-certs\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.193959 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.193859 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-dshm\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.294432 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.294398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-tmp-dir\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.294569 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.294437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-home\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.294569 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.294484 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-model-cache\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.294569 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.294516 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lgjv\" (UniqueName: \"kubernetes.io/projected/9b2856ae-9b61-40c6-9738-de7425f39456-kube-api-access-8lgjv\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.294569 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.294540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b2856ae-9b61-40c6-9738-de7425f39456-tls-certs\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.294771 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.294581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-dshm\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.294853 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.294831 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-tmp-dir\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.294912 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.294860 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-home\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.294912 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.294903 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-model-cache\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.296846 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.296827 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-dshm\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.297000 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.296983 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b2856ae-9b61-40c6-9738-de7425f39456-tls-certs\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.302985 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.302968 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lgjv\" (UniqueName: \"kubernetes.io/projected/9b2856ae-9b61-40c6-9738-de7425f39456-kube-api-access-8lgjv\") pod \"test-model-6c210e-kserve-79965b8d59-mjzss\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.429108 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.429048 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:28.559003 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.558981 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss"] Aug 04 16:44:28.564833 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:44:28.564807 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b2856ae_9b61_40c6_9738_de7425f39456.slice/crio-53f58ec3bd58437304f35d6a6c16d8ea2a3efcb50066c1902f46a4c130325182 WatchSource:0}: Error finding container 53f58ec3bd58437304f35d6a6c16d8ea2a3efcb50066c1902f46a4c130325182: Status 404 returned error can't find the container with id 53f58ec3bd58437304f35d6a6c16d8ea2a3efcb50066c1902f46a4c130325182 Aug 04 16:44:28.795243 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.795163 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" event={"ID":"9b2856ae-9b61-40c6-9738-de7425f39456","Type":"ContainerStarted","Data":"61f25e0baca30be2bc901b229c882a71083babbdaf19421ec53ad22b32034689"} Aug 04 16:44:28.795243 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.795203 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" event={"ID":"9b2856ae-9b61-40c6-9738-de7425f39456","Type":"ContainerStarted","Data":"53f58ec3bd58437304f35d6a6c16d8ea2a3efcb50066c1902f46a4c130325182"} Aug 04 16:44:28.813009 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:28.812964 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" podStartSLOduration=0.812949755 podStartE2EDuration="812.949755ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:44:28.812004179 +0000 UTC m=+2569.956336424" watchObservedRunningTime="2026-08-04 16:44:28.812949755 +0000 UTC m=+2569.957282001" Aug 04 16:44:30.438230 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:30.438199 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:44:38.430198 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:38.430169 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:38.430697 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:38.430214 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:38.442001 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:38.441976 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:44:38.633924 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:38.633897 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:44:38.853681 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:44:38.853650 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:45:00.132701 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.132671 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764365-bgb2k"] Aug 04 16:45:00.135909 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.135887 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-bgb2k" Aug 04 16:45:00.138752 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.138731 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19c94c-29764365-jg2jp"] Aug 04 16:45:00.138920 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.138900 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4pxfk\"" Aug 04 16:45:00.141814 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.141796 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6c210e-29764365-zkgct"] Aug 04 16:45:00.142133 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.141918 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19c94c-29764365-jg2jp" Aug 04 16:45:00.145553 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.145531 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764365-bgb2k"] Aug 04 16:45:00.145654 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.145623 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6c210e-29764365-zkgct" Aug 04 16:45:00.147583 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.147563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-bgb2k" Aug 04 16:45:00.154091 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.153655 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19c94c-29764365-jg2jp"] Aug 04 16:45:00.154091 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.154016 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6c210e-29764365-zkgct"] Aug 04 16:45:00.160064 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.160043 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19c94c-29764365-jg2jp" Aug 04 16:45:00.195354 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.195330 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6c210e-29764365-zkgct" Aug 04 16:45:00.313716 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.313665 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764365-bgb2k"] Aug 04 16:45:00.317649 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:45:00.316809 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9fd05b_a465_465b_b4b8_707e0ea5d4ae.slice/crio-f01017effd28737367d7d2d363279bd7517c8a54281649f6e51eaca9ab50e8fe WatchSource:0}: Error finding container f01017effd28737367d7d2d363279bd7517c8a54281649f6e51eaca9ab50e8fe: Status 404 returned error can't find the container with id f01017effd28737367d7d2d363279bd7517c8a54281649f6e51eaca9ab50e8fe Aug 04 16:45:00.548856 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.548830 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19c94c-29764365-jg2jp"] Aug 04 16:45:00.558384 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:45:00.558357 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0abb144_5aa3_443c_8c2a_ad1ad919776f.slice/crio-57dc3b5bb3858ea10006946d8564a96ad4278d50a3711e2479694dd9f25dae1f WatchSource:0}: Error finding container 57dc3b5bb3858ea10006946d8564a96ad4278d50a3711e2479694dd9f25dae1f: Status 404 returned error can't find the container with id 57dc3b5bb3858ea10006946d8564a96ad4278d50a3711e2479694dd9f25dae1f Aug 04 16:45:00.572928 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.572886 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6c210e-29764365-zkgct"] Aug 04 16:45:00.574763 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:45:00.574740 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c5ca4f_78af_4459_9e21_2bedf340e32c.slice/crio-fe343e72cd14cc0c3a38ce199f1c8948d9fa9ea647b7ee9d69c29138f8ec1d3f WatchSource:0}: Error finding container fe343e72cd14cc0c3a38ce199f1c8948d9fa9ea647b7ee9d69c29138f8ec1d3f: Status 404 returned error can't find the container with id fe343e72cd14cc0c3a38ce199f1c8948d9fa9ea647b7ee9d69c29138f8ec1d3f Aug 04 16:45:00.935715 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.935682 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-bgb2k" event={"ID":"0a9fd05b-a465-465b-b4b8-707e0ea5d4ae","Type":"ContainerStarted","Data":"2a4f371faccba0223f1751bc82f73dcb810bdb725544bdf833e441ce86513312"} Aug 04 16:45:00.935857 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.935742 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-bgb2k" event={"ID":"0a9fd05b-a465-465b-b4b8-707e0ea5d4ae","Type":"ContainerStarted","Data":"f01017effd28737367d7d2d363279bd7517c8a54281649f6e51eaca9ab50e8fe"} Aug 04 16:45:00.937221 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.937191 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19c94c-29764365-jg2jp" event={"ID":"d0abb144-5aa3-443c-8c2a-ad1ad919776f","Type":"ContainerStarted","Data":"067af14d2dc88489b37a4af31d547055085597c4a423278d17d9ee566535ad7a"} Aug 04 16:45:00.937334 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.937249 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19c94c-29764365-jg2jp" event={"ID":"d0abb144-5aa3-443c-8c2a-ad1ad919776f","Type":"ContainerStarted","Data":"57dc3b5bb3858ea10006946d8564a96ad4278d50a3711e2479694dd9f25dae1f"} Aug 04 16:45:00.938612 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.938590 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6c210e-29764365-zkgct" event={"ID":"05c5ca4f-78af-4459-9e21-2bedf340e32c","Type":"ContainerStarted","Data":"53d270eaf05ef081dd639fa1c5084559f8906ee60170d706f7cd49fe3730bd00"} Aug 04 16:45:00.938681 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.938628 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6c210e-29764365-zkgct" event={"ID":"05c5ca4f-78af-4459-9e21-2bedf340e32c","Type":"ContainerStarted","Data":"fe343e72cd14cc0c3a38ce199f1c8948d9fa9ea647b7ee9d69c29138f8ec1d3f"} Aug 04 16:45:00.954401 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.954357 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-bgb2k" podStartSLOduration=0.954344025 podStartE2EDuration="954.344025ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:45:00.953990545 +0000 UTC m=+2602.098322809" watchObservedRunningTime="2026-08-04 16:45:00.954344025 +0000 UTC m=+2602.098676270" Aug 04 16:45:00.969577 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.969538 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19c94c-29764365-jg2jp" podStartSLOduration=0.969525002 podStartE2EDuration="969.525002ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:45:00.968245919 +0000 UTC m=+2602.112578183" watchObservedRunningTime="2026-08-04 16:45:00.969525002 +0000 UTC m=+2602.113857261" Aug 04 16:45:00.984169 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:00.984135 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6c210e-29764365-zkgct" podStartSLOduration=0.984126794 podStartE2EDuration="984.126794ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:45:00.983473812 +0000 UTC m=+2602.127806082" watchObservedRunningTime="2026-08-04 16:45:00.984126794 +0000 UTC m=+2602.128459040" Aug 04 16:45:01.152524 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:01.152454 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:45:01.947466 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:01.947431 2580 generic.go:356] "Generic (PLEG): container finished" podID="d0abb144-5aa3-443c-8c2a-ad1ad919776f" containerID="067af14d2dc88489b37a4af31d547055085597c4a423278d17d9ee566535ad7a" exitCode=0 Aug 04 16:45:01.947629 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:01.947520 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19c94c-29764365-jg2jp" event={"ID":"d0abb144-5aa3-443c-8c2a-ad1ad919776f","Type":"ContainerDied","Data":"067af14d2dc88489b37a4af31d547055085597c4a423278d17d9ee566535ad7a"} Aug 04 16:45:01.948869 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:01.948848 2580 generic.go:356] "Generic (PLEG): container finished" podID="05c5ca4f-78af-4459-9e21-2bedf340e32c" containerID="53d270eaf05ef081dd639fa1c5084559f8906ee60170d706f7cd49fe3730bd00" exitCode=0 Aug 04 16:45:01.948956 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:01.948905 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6c210e-29764365-zkgct" event={"ID":"05c5ca4f-78af-4459-9e21-2bedf340e32c","Type":"ContainerDied","Data":"53d270eaf05ef081dd639fa1c5084559f8906ee60170d706f7cd49fe3730bd00"} Aug 04 16:45:01.950570 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:01.950550 2580 generic.go:356] "Generic (PLEG): container finished" podID="0a9fd05b-a465-465b-b4b8-707e0ea5d4ae" containerID="2a4f371faccba0223f1751bc82f73dcb810bdb725544bdf833e441ce86513312" exitCode=0 Aug 04 16:45:01.950661 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:01.950583 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-bgb2k" event={"ID":"0a9fd05b-a465-465b-b4b8-707e0ea5d4ae","Type":"ContainerDied","Data":"2a4f371faccba0223f1751bc82f73dcb810bdb725544bdf833e441ce86513312"} Aug 04 16:45:03.131634 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:03.131612 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-bgb2k" Aug 04 16:45:03.134635 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:03.134615 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6c210e-29764365-zkgct" Aug 04 16:45:03.137564 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:03.137545 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19c94c-29764365-jg2jp" Aug 04 16:45:03.961511 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:03.961476 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-bgb2k" Aug 04 16:45:03.961511 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:03.961476 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-bgb2k" event={"ID":"0a9fd05b-a465-465b-b4b8-707e0ea5d4ae","Type":"ContainerDied","Data":"f01017effd28737367d7d2d363279bd7517c8a54281649f6e51eaca9ab50e8fe"} Aug 04 16:45:03.961715 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:03.961537 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f01017effd28737367d7d2d363279bd7517c8a54281649f6e51eaca9ab50e8fe" Aug 04 16:45:03.962843 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:03.962826 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19c94c-29764365-jg2jp" Aug 04 16:45:03.962843 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:03.962830 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19c94c-29764365-jg2jp" event={"ID":"d0abb144-5aa3-443c-8c2a-ad1ad919776f","Type":"ContainerDied","Data":"57dc3b5bb3858ea10006946d8564a96ad4278d50a3711e2479694dd9f25dae1f"} Aug 04 16:45:03.963004 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:03.962872 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57dc3b5bb3858ea10006946d8564a96ad4278d50a3711e2479694dd9f25dae1f" Aug 04 16:45:03.964234 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:03.964217 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6c210e-29764365-zkgct" Aug 04 16:45:03.964234 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:03.964226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6c210e-29764365-zkgct" event={"ID":"05c5ca4f-78af-4459-9e21-2bedf340e32c","Type":"ContainerDied","Data":"fe343e72cd14cc0c3a38ce199f1c8948d9fa9ea647b7ee9d69c29138f8ec1d3f"} Aug 04 16:45:03.964391 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:03.964262 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe343e72cd14cc0c3a38ce199f1c8948d9fa9ea647b7ee9d69c29138f8ec1d3f" Aug 04 16:45:12.139181 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:12.139146 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:45:40.493688 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:40.493652 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss"] Aug 04 16:45:41.586220 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.586191 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5"] Aug 04 16:45:41.586596 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.586559 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c5ca4f-78af-4459-9e21-2bedf340e32c" containerName="cleanup" Aug 04 16:45:41.586596 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.586572 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c5ca4f-78af-4459-9e21-2bedf340e32c" containerName="cleanup" Aug 04 16:45:41.586596 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.586586 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9fd05b-a465-465b-b4b8-707e0ea5d4ae" containerName="cleanup" Aug 04 16:45:41.586596 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.586592 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9fd05b-a465-465b-b4b8-707e0ea5d4ae" containerName="cleanup" Aug 04 16:45:41.586596 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.586599 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0abb144-5aa3-443c-8c2a-ad1ad919776f" containerName="cleanup" Aug 04 16:45:41.586753 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.586605 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0abb144-5aa3-443c-8c2a-ad1ad919776f" containerName="cleanup" Aug 04 16:45:41.586753 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.586682 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0abb144-5aa3-443c-8c2a-ad1ad919776f" containerName="cleanup" Aug 04 16:45:41.586753 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.586691 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a9fd05b-a465-465b-b4b8-707e0ea5d4ae" containerName="cleanup" Aug 04 16:45:41.586753 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.586697 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="05c5ca4f-78af-4459-9e21-2bedf340e32c" containerName="cleanup" Aug 04 16:45:41.590026 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.590008 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5" Aug 04 16:45:41.592860 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.592839 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4pxfk\"" Aug 04 16:45:41.592992 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.592859 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:45:41.598037 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.598009 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5"] Aug 04 16:45:41.652707 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.652672 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5\" (UID: \"1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5" Aug 04 16:45:41.753370 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.753336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5\" (UID: \"1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5" Aug 04 16:45:41.753967 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.753940 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5\" (UID: \"1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5" Aug 04 16:45:41.901999 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:41.901923 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5" Aug 04 16:45:42.029219 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:42.029195 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5"] Aug 04 16:45:42.031339 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:45:42.031307 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec7a772_b27a_4fed_a4ea_f5d8cbe6c785.slice/crio-d2eb939aa90a59112618fc79429235db96cfd34dab50de502593fb07b5262545 WatchSource:0}: Error finding container d2eb939aa90a59112618fc79429235db96cfd34dab50de502593fb07b5262545: Status 404 returned error can't find the container with id d2eb939aa90a59112618fc79429235db96cfd34dab50de502593fb07b5262545 Aug 04 16:45:42.144341 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:42.144310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5" event={"ID":"1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785","Type":"ContainerStarted","Data":"d2eb939aa90a59112618fc79429235db96cfd34dab50de502593fb07b5262545"} Aug 04 16:45:43.150164 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:43.150135 2580 generic.go:356] "Generic (PLEG): container finished" podID="1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785" containerID="1a61e49b13bb7b418508c985e4f9e6618375874b046b981e00f12e0c55625c5f" exitCode=0 Aug 04 16:45:43.150527 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:43.150224 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5" event={"ID":"1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785","Type":"ContainerDied","Data":"1a61e49b13bb7b418508c985e4f9e6618375874b046b981e00f12e0c55625c5f"} Aug 04 16:45:44.285369 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:44.285342 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5" Aug 04 16:45:44.373893 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:44.373870 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785-maas-api-service-ca\") pod \"1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785\" (UID: \"1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785\") " Aug 04 16:45:44.374191 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:44.374170 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785" (UID: "1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:45:44.474694 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:44.474635 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785-maas-api-service-ca\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:45:45.160464 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:45.160435 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5" Aug 04 16:45:45.160618 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:45.160431 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5" event={"ID":"1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785","Type":"ContainerDied","Data":"d2eb939aa90a59112618fc79429235db96cfd34dab50de502593fb07b5262545"} Aug 04 16:45:45.160618 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:45.160554 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2eb939aa90a59112618fc79429235db96cfd34dab50de502593fb07b5262545" Aug 04 16:45:45.536429 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:45.536352 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:45:52.207297 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:52.207250 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6c210e-29764365-zkgct"] Aug 04 16:45:52.209363 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:52.209341 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6c210e-29764365-zkgct"] Aug 04 16:45:52.403124 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:52.403090 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5"] Aug 04 16:45:52.406079 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:52.406052 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6c210e-cxpk5"] Aug 04 16:45:53.318795 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:53.318756 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05c5ca4f-78af-4459-9e21-2bedf340e32c" path="/var/lib/kubelet/pods/05c5ca4f-78af-4459-9e21-2bedf340e32c/volumes" Aug 04 16:45:53.319168 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:53.319099 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785" path="/var/lib/kubelet/pods/1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785/volumes" Aug 04 16:45:54.433508 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:54.433473 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:45:55.511976 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.511926 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" podUID="9b2856ae-9b61-40c6-9738-de7425f39456" containerName="main" containerID="cri-o://61f25e0baca30be2bc901b229c882a71083babbdaf19421ec53ad22b32034689" gracePeriod=45 Aug 04 16:45:55.754091 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.754069 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:45:55.855357 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.855334 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lgjv\" (UniqueName: \"kubernetes.io/projected/9b2856ae-9b61-40c6-9738-de7425f39456-kube-api-access-8lgjv\") pod \"9b2856ae-9b61-40c6-9738-de7425f39456\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " Aug 04 16:45:55.855476 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.855401 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-home\") pod \"9b2856ae-9b61-40c6-9738-de7425f39456\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " Aug 04 16:45:55.855476 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.855416 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-dshm\") pod \"9b2856ae-9b61-40c6-9738-de7425f39456\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " Aug 04 16:45:55.855550 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.855531 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b2856ae-9b61-40c6-9738-de7425f39456-tls-certs\") pod \"9b2856ae-9b61-40c6-9738-de7425f39456\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " Aug 04 16:45:55.855583 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.855570 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-tmp-dir\") pod \"9b2856ae-9b61-40c6-9738-de7425f39456\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " Aug 04 16:45:55.855621 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.855601 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-model-cache\") pod \"9b2856ae-9b61-40c6-9738-de7425f39456\" (UID: \"9b2856ae-9b61-40c6-9738-de7425f39456\") " Aug 04 16:45:55.855773 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.855692 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-home" (OuterVolumeSpecName: "home") pod "9b2856ae-9b61-40c6-9738-de7425f39456" (UID: "9b2856ae-9b61-40c6-9738-de7425f39456"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:45:55.856085 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.855857 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b2856ae-9b61-40c6-9738-de7425f39456" (UID: "9b2856ae-9b61-40c6-9738-de7425f39456"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:45:55.856085 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.855889 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-home\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:45:55.856085 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.855896 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b2856ae-9b61-40c6-9738-de7425f39456" (UID: "9b2856ae-9b61-40c6-9738-de7425f39456"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:45:55.857625 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.857602 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b2856ae-9b61-40c6-9738-de7425f39456-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b2856ae-9b61-40c6-9738-de7425f39456" (UID: "9b2856ae-9b61-40c6-9738-de7425f39456"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:45:55.857920 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.857900 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2856ae-9b61-40c6-9738-de7425f39456-kube-api-access-8lgjv" (OuterVolumeSpecName: "kube-api-access-8lgjv") pod "9b2856ae-9b61-40c6-9738-de7425f39456" (UID: "9b2856ae-9b61-40c6-9738-de7425f39456"). InnerVolumeSpecName "kube-api-access-8lgjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:45:55.857920 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.857907 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-dshm" (OuterVolumeSpecName: "dshm") pod "9b2856ae-9b61-40c6-9738-de7425f39456" (UID: "9b2856ae-9b61-40c6-9738-de7425f39456"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:45:55.956989 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.956954 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-dshm\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:45:55.956989 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.956987 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b2856ae-9b61-40c6-9738-de7425f39456-tls-certs\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:45:55.957126 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.957003 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-tmp-dir\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:45:55.957126 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.957013 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b2856ae-9b61-40c6-9738-de7425f39456-model-cache\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:45:55.957126 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:55.957024 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lgjv\" (UniqueName: \"kubernetes.io/projected/9b2856ae-9b61-40c6-9738-de7425f39456-kube-api-access-8lgjv\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:45:56.219771 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:56.219698 2580 generic.go:356] "Generic (PLEG): container finished" podID="9b2856ae-9b61-40c6-9738-de7425f39456" containerID="61f25e0baca30be2bc901b229c882a71083babbdaf19421ec53ad22b32034689" exitCode=0 Aug 04 16:45:56.219771 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:56.219755 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" Aug 04 16:45:56.219948 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:56.219772 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" event={"ID":"9b2856ae-9b61-40c6-9738-de7425f39456","Type":"ContainerDied","Data":"61f25e0baca30be2bc901b229c882a71083babbdaf19421ec53ad22b32034689"} Aug 04 16:45:56.219948 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:56.219812 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss" event={"ID":"9b2856ae-9b61-40c6-9738-de7425f39456","Type":"ContainerDied","Data":"53f58ec3bd58437304f35d6a6c16d8ea2a3efcb50066c1902f46a4c130325182"} Aug 04 16:45:56.219948 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:56.219834 2580 scope.go:117] "RemoveContainer" containerID="61f25e0baca30be2bc901b229c882a71083babbdaf19421ec53ad22b32034689" Aug 04 16:45:56.229027 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:56.229013 2580 scope.go:117] "RemoveContainer" containerID="61f25e0baca30be2bc901b229c882a71083babbdaf19421ec53ad22b32034689" Aug 04 16:45:56.229273 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:45:56.229256 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f25e0baca30be2bc901b229c882a71083babbdaf19421ec53ad22b32034689\": container with ID starting with 61f25e0baca30be2bc901b229c882a71083babbdaf19421ec53ad22b32034689 not found: ID does not exist" containerID="61f25e0baca30be2bc901b229c882a71083babbdaf19421ec53ad22b32034689" Aug 04 16:45:56.229332 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:56.229302 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f25e0baca30be2bc901b229c882a71083babbdaf19421ec53ad22b32034689"} err="failed to get container status \"61f25e0baca30be2bc901b229c882a71083babbdaf19421ec53ad22b32034689\": rpc error: code = NotFound desc = could not find container \"61f25e0baca30be2bc901b229c882a71083babbdaf19421ec53ad22b32034689\": container with ID starting with 61f25e0baca30be2bc901b229c882a71083babbdaf19421ec53ad22b32034689 not found: ID does not exist" Aug 04 16:45:56.241905 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:56.241882 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss"] Aug 04 16:45:56.247175 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:56.247155 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6c210e/test-model-6c210e-kserve-79965b8d59-mjzss"] Aug 04 16:45:57.319121 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:57.319089 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b2856ae-9b61-40c6-9738-de7425f39456" path="/var/lib/kubelet/pods/9b2856ae-9b61-40c6-9738-de7425f39456/volumes" Aug 04 16:45:59.621593 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:59.621556 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb"] Aug 04 16:45:59.622096 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:59.622076 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b2856ae-9b61-40c6-9738-de7425f39456" containerName="main" Aug 04 16:45:59.622175 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:59.622099 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b2856ae-9b61-40c6-9738-de7425f39456" containerName="main" Aug 04 16:45:59.622175 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:59.622112 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785" containerName="revoke-keys" Aug 04 16:45:59.622175 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:59.622122 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785" containerName="revoke-keys" Aug 04 16:45:59.622360 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:59.622207 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ec7a772-b27a-4fed-a4ea-f5d8cbe6c785" containerName="revoke-keys" Aug 04 16:45:59.622360 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:59.622226 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b2856ae-9b61-40c6-9738-de7425f39456" containerName="main" Aug 04 16:45:59.626578 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:59.626558 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb" Aug 04 16:45:59.629463 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:59.629441 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:45:59.629633 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:59.629447 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4pxfk\"" Aug 04 16:45:59.633227 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:59.633208 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb"] Aug 04 16:45:59.690303 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:59.690259 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8bfdad-d293-4758-a06b-df2ae3a3d675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb\" (UID: \"2d8bfdad-d293-4758-a06b-df2ae3a3d675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb" Aug 04 16:45:59.791324 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:59.791276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8bfdad-d293-4758-a06b-df2ae3a3d675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb\" (UID: \"2d8bfdad-d293-4758-a06b-df2ae3a3d675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb" Aug 04 16:45:59.791871 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:59.791846 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8bfdad-d293-4758-a06b-df2ae3a3d675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb\" (UID: \"2d8bfdad-d293-4758-a06b-df2ae3a3d675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb" Aug 04 16:45:59.937697 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:45:59.937634 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb" Aug 04 16:46:00.063940 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:00.063915 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb"] Aug 04 16:46:00.066682 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:46:00.066651 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8bfdad_d293_4758_a06b_df2ae3a3d675.slice/crio-c8424a2469e802402ae98bb2ace337ff7806c7761e97a70988100afbc06c2e9a WatchSource:0}: Error finding container c8424a2469e802402ae98bb2ace337ff7806c7761e97a70988100afbc06c2e9a: Status 404 returned error can't find the container with id c8424a2469e802402ae98bb2ace337ff7806c7761e97a70988100afbc06c2e9a Aug 04 16:46:00.239964 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:00.239899 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb" event={"ID":"2d8bfdad-d293-4758-a06b-df2ae3a3d675","Type":"ContainerStarted","Data":"c8424a2469e802402ae98bb2ace337ff7806c7761e97a70988100afbc06c2e9a"} Aug 04 16:46:01.246641 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:01.246606 2580 generic.go:356] "Generic (PLEG): container finished" podID="2d8bfdad-d293-4758-a06b-df2ae3a3d675" containerID="8df7c503e23d718984cc9943a584e3afc2036a39ba7e324e59a92c79837876ed" exitCode=0 Aug 04 16:46:01.247018 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:01.246700 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb" event={"ID":"2d8bfdad-d293-4758-a06b-df2ae3a3d675","Type":"ContainerDied","Data":"8df7c503e23d718984cc9943a584e3afc2036a39ba7e324e59a92c79837876ed"} Aug 04 16:46:02.385952 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:02.385929 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb" Aug 04 16:46:02.515334 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:02.515244 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8bfdad-d293-4758-a06b-df2ae3a3d675-maas-api-service-ca\") pod \"2d8bfdad-d293-4758-a06b-df2ae3a3d675\" (UID: \"2d8bfdad-d293-4758-a06b-df2ae3a3d675\") " Aug 04 16:46:02.515583 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:02.515558 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8bfdad-d293-4758-a06b-df2ae3a3d675-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d8bfdad-d293-4758-a06b-df2ae3a3d675" (UID: "2d8bfdad-d293-4758-a06b-df2ae3a3d675"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:46:02.615887 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:02.615863 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8bfdad-d293-4758-a06b-df2ae3a3d675-maas-api-service-ca\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:46:03.259288 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:03.259252 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb" Aug 04 16:46:03.259430 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:03.259299 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb" event={"ID":"2d8bfdad-d293-4758-a06b-df2ae3a3d675","Type":"ContainerDied","Data":"c8424a2469e802402ae98bb2ace337ff7806c7761e97a70988100afbc06c2e9a"} Aug 04 16:46:03.259430 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:03.259375 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8424a2469e802402ae98bb2ace337ff7806c7761e97a70988100afbc06c2e9a" Aug 04 16:46:10.208149 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:10.208117 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19c94c-29764365-jg2jp"] Aug 04 16:46:10.209856 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:10.209835 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19c94c-29764365-jg2jp"] Aug 04 16:46:10.374784 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:10.374747 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb"] Aug 04 16:46:10.376293 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:10.376255 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19c94c-jq6jb"] Aug 04 16:46:11.318832 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:11.318801 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d8bfdad-d293-4758-a06b-df2ae3a3d675" path="/var/lib/kubelet/pods/2d8bfdad-d293-4758-a06b-df2ae3a3d675/volumes" Aug 04 16:46:11.319211 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:11.319070 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0abb144-5aa3-443c-8c2a-ad1ad919776f" path="/var/lib/kubelet/pods/d0abb144-5aa3-443c-8c2a-ad1ad919776f/volumes" Aug 04 16:46:12.444337 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:12.444307 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:46:24.433084 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:24.433051 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:46:39.411829 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:39.411755 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:46:39.419273 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:39.419252 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:46:50.805124 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:50.805090 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8"] Aug 04 16:46:50.805564 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:50.805477 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d8bfdad-d293-4758-a06b-df2ae3a3d675" containerName="revoke-keys" Aug 04 16:46:50.805564 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:50.805490 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d8bfdad-d293-4758-a06b-df2ae3a3d675" containerName="revoke-keys" Aug 04 16:46:50.805564 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:50.805553 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d8bfdad-d293-4758-a06b-df2ae3a3d675" containerName="revoke-keys" Aug 04 16:46:50.808290 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:50.808264 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" Aug 04 16:46:50.811138 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:50.811116 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2847c7-dockercfg-lcmjm\"" Aug 04 16:46:50.811138 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:50.811135 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2847c7\"" Aug 04 16:46:50.818243 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:50.818221 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8"] Aug 04 16:46:50.972672 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:50.972643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcpd6\" (UniqueName: \"kubernetes.io/projected/729a4559-9a57-4e05-9068-acce71cc9519-kube-api-access-gcpd6\") pod \"payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8\" (UID: \"729a4559-9a57-4e05-9068-acce71cc9519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" Aug 04 16:46:50.972796 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:50.972687 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729a4559-9a57-4e05-9068-acce71cc9519-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8\" (UID: \"729a4559-9a57-4e05-9068-acce71cc9519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" Aug 04 16:46:51.073354 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:51.073327 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729a4559-9a57-4e05-9068-acce71cc9519-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8\" (UID: \"729a4559-9a57-4e05-9068-acce71cc9519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" Aug 04 16:46:51.073480 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:51.073435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcpd6\" (UniqueName: \"kubernetes.io/projected/729a4559-9a57-4e05-9068-acce71cc9519-kube-api-access-gcpd6\") pod \"payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8\" (UID: \"729a4559-9a57-4e05-9068-acce71cc9519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" Aug 04 16:46:51.073978 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:51.073956 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729a4559-9a57-4e05-9068-acce71cc9519-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8\" (UID: \"729a4559-9a57-4e05-9068-acce71cc9519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" Aug 04 16:46:51.082028 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:51.082006 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcpd6\" (UniqueName: \"kubernetes.io/projected/729a4559-9a57-4e05-9068-acce71cc9519-kube-api-access-gcpd6\") pod \"payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8\" (UID: \"729a4559-9a57-4e05-9068-acce71cc9519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" Aug 04 16:46:51.119892 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:51.119869 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" Aug 04 16:46:51.246885 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:51.246821 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8"] Aug 04 16:46:51.249916 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:46:51.249879 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729a4559_9a57_4e05_9068_acce71cc9519.slice/crio-ecb3c3af71ad9315256b38bbfdf2b8ecf83bae792d7a3dd7ca3c4805e4327aac WatchSource:0}: Error finding container ecb3c3af71ad9315256b38bbfdf2b8ecf83bae792d7a3dd7ca3c4805e4327aac: Status 404 returned error can't find the container with id ecb3c3af71ad9315256b38bbfdf2b8ecf83bae792d7a3dd7ca3c4805e4327aac Aug 04 16:46:51.252456 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:51.252438 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:46:51.482630 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:51.482559 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" event={"ID":"729a4559-9a57-4e05-9068-acce71cc9519","Type":"ContainerStarted","Data":"ecb3c3af71ad9315256b38bbfdf2b8ecf83bae792d7a3dd7ca3c4805e4327aac"} Aug 04 16:46:53.139907 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:53.139872 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:46:54.499701 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:54.499661 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" event={"ID":"729a4559-9a57-4e05-9068-acce71cc9519","Type":"ContainerStarted","Data":"742217a6d04d0b3444573b5cbc6e12313f5492620c27303e37a17e1325e0aba5"} Aug 04 16:46:54.500079 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:54.499758 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" Aug 04 16:46:54.517123 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:46:54.517076 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" podStartSLOduration=2.248450374 podStartE2EDuration="4.51706216s" totalImagesPullingTime="2.268611786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:46:54.515606998 +0000 UTC m=+2715.659939264" watchObservedRunningTime="2026-08-04 16:46:54.51706216 +0000 UTC m=+2715.661394405" Aug 04 16:47:05.506532 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:05.506503 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" Aug 04 16:47:16.974407 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:16.974375 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-734f56-l44bn"] Aug 04 16:47:16.979624 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:16.979602 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-734f56-l44bn" Aug 04 16:47:16.982421 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:16.982402 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:47:16.982541 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:16.982401 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4pxfk\"" Aug 04 16:47:16.986669 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:16.986620 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-734f56-l44bn"] Aug 04 16:47:17.159727 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:17.159703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c518b2db-0309-4bfa-a6ba-257f8fddb74e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-734f56-l44bn\" (UID: \"c518b2db-0309-4bfa-a6ba-257f8fddb74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-734f56-l44bn" Aug 04 16:47:17.260781 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:17.260712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c518b2db-0309-4bfa-a6ba-257f8fddb74e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-734f56-l44bn\" (UID: \"c518b2db-0309-4bfa-a6ba-257f8fddb74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-734f56-l44bn" Aug 04 16:47:17.261251 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:17.261234 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c518b2db-0309-4bfa-a6ba-257f8fddb74e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-734f56-l44bn\" (UID: \"c518b2db-0309-4bfa-a6ba-257f8fddb74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-734f56-l44bn" Aug 04 16:47:17.290944 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:17.290918 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-734f56-l44bn" Aug 04 16:47:17.418678 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:17.418654 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-734f56-l44bn"] Aug 04 16:47:17.421054 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:47:17.421021 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc518b2db_0309_4bfa_a6ba_257f8fddb74e.slice/crio-a3134415dda91f5b8989800316bbb1b43a31a8becb8dfad69e756005c781f7be WatchSource:0}: Error finding container a3134415dda91f5b8989800316bbb1b43a31a8becb8dfad69e756005c781f7be: Status 404 returned error can't find the container with id a3134415dda91f5b8989800316bbb1b43a31a8becb8dfad69e756005c781f7be Aug 04 16:47:17.609947 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:17.609918 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-734f56-l44bn" event={"ID":"c518b2db-0309-4bfa-a6ba-257f8fddb74e","Type":"ContainerStarted","Data":"a3134415dda91f5b8989800316bbb1b43a31a8becb8dfad69e756005c781f7be"} Aug 04 16:47:18.616081 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:18.616048 2580 generic.go:356] "Generic (PLEG): container finished" podID="c518b2db-0309-4bfa-a6ba-257f8fddb74e" containerID="ee99c09b8a5f0af4ed410289eb3e1a9dd7d3646f87d211f30056eb0b18a4566c" exitCode=0 Aug 04 16:47:18.616456 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:18.616128 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-734f56-l44bn" event={"ID":"c518b2db-0309-4bfa-a6ba-257f8fddb74e","Type":"ContainerDied","Data":"ee99c09b8a5f0af4ed410289eb3e1a9dd7d3646f87d211f30056eb0b18a4566c"} Aug 04 16:47:19.747512 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:19.747483 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-734f56-l44bn" Aug 04 16:47:19.779830 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:19.779808 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c518b2db-0309-4bfa-a6ba-257f8fddb74e-maas-api-service-ca\") pod \"c518b2db-0309-4bfa-a6ba-257f8fddb74e\" (UID: \"c518b2db-0309-4bfa-a6ba-257f8fddb74e\") " Aug 04 16:47:19.780143 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:19.780112 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c518b2db-0309-4bfa-a6ba-257f8fddb74e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c518b2db-0309-4bfa-a6ba-257f8fddb74e" (UID: "c518b2db-0309-4bfa-a6ba-257f8fddb74e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:47:19.880920 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:19.880867 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c518b2db-0309-4bfa-a6ba-257f8fddb74e-maas-api-service-ca\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:47:20.626318 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:20.626291 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-734f56-l44bn" Aug 04 16:47:20.626484 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:20.626313 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-734f56-l44bn" event={"ID":"c518b2db-0309-4bfa-a6ba-257f8fddb74e","Type":"ContainerDied","Data":"a3134415dda91f5b8989800316bbb1b43a31a8becb8dfad69e756005c781f7be"} Aug 04 16:47:20.626484 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:20.626362 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3134415dda91f5b8989800316bbb1b43a31a8becb8dfad69e756005c781f7be" Aug 04 16:47:32.601947 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:32.601912 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-734f56-l44bn"] Aug 04 16:47:32.606354 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:32.606331 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-734f56-l44bn"] Aug 04 16:47:33.319561 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:33.319531 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c518b2db-0309-4bfa-a6ba-257f8fddb74e" path="/var/lib/kubelet/pods/c518b2db-0309-4bfa-a6ba-257f8fddb74e/volumes" Aug 04 16:47:34.934266 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:34.934235 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:47:39.424026 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:39.423993 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9"] Aug 04 16:47:39.424556 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:39.424506 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c518b2db-0309-4bfa-a6ba-257f8fddb74e" containerName="revoke-keys" Aug 04 16:47:39.424556 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:39.424526 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c518b2db-0309-4bfa-a6ba-257f8fddb74e" containerName="revoke-keys" Aug 04 16:47:39.424673 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:39.424640 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c518b2db-0309-4bfa-a6ba-257f8fddb74e" containerName="revoke-keys" Aug 04 16:47:39.427803 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:39.427783 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9" Aug 04 16:47:39.430659 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:39.430641 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:47:39.430810 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:39.430795 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4pxfk\"" Aug 04 16:47:39.434800 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:39.434610 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9"] Aug 04 16:47:39.532756 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:39.532726 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66dd82e6-51ac-4502-8c83-6befbb446cff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9\" (UID: \"66dd82e6-51ac-4502-8c83-6befbb446cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9" Aug 04 16:47:39.546750 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:39.546729 2580 scope.go:117] "RemoveContainer" containerID="bca19fbe3bfd766fc50881d5b75aca2ed51d2e71351f5c739569ec77b3f5125b" Aug 04 16:47:39.633531 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:39.633499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66dd82e6-51ac-4502-8c83-6befbb446cff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9\" (UID: \"66dd82e6-51ac-4502-8c83-6befbb446cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9" Aug 04 16:47:39.634029 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:39.634012 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66dd82e6-51ac-4502-8c83-6befbb446cff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9\" (UID: \"66dd82e6-51ac-4502-8c83-6befbb446cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9" Aug 04 16:47:39.738929 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:39.738873 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9" Aug 04 16:47:40.074226 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:40.072670 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9"] Aug 04 16:47:40.076732 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:47:40.076700 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66dd82e6_51ac_4502_8c83_6befbb446cff.slice/crio-a909a7e9ab61b4ab246ee86168bf94ea7e54ce684f1cd3a5ea2087de80455d00 WatchSource:0}: Error finding container a909a7e9ab61b4ab246ee86168bf94ea7e54ce684f1cd3a5ea2087de80455d00: Status 404 returned error can't find the container with id a909a7e9ab61b4ab246ee86168bf94ea7e54ce684f1cd3a5ea2087de80455d00 Aug 04 16:47:40.720306 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:40.720264 2580 generic.go:356] "Generic (PLEG): container finished" podID="66dd82e6-51ac-4502-8c83-6befbb446cff" containerID="18cfbd386280b4c5c56fdbc2c0a9b6a8fed485c50eb51e9c81545dda64cbd239" exitCode=0 Aug 04 16:47:40.720654 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:40.720329 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9" event={"ID":"66dd82e6-51ac-4502-8c83-6befbb446cff","Type":"ContainerDied","Data":"18cfbd386280b4c5c56fdbc2c0a9b6a8fed485c50eb51e9c81545dda64cbd239"} Aug 04 16:47:40.720654 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:40.720404 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9" event={"ID":"66dd82e6-51ac-4502-8c83-6befbb446cff","Type":"ContainerStarted","Data":"a909a7e9ab61b4ab246ee86168bf94ea7e54ce684f1cd3a5ea2087de80455d00"} Aug 04 16:47:41.853623 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:41.853599 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9" Aug 04 16:47:41.954669 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:41.954643 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66dd82e6-51ac-4502-8c83-6befbb446cff-maas-api-service-ca\") pod \"66dd82e6-51ac-4502-8c83-6befbb446cff\" (UID: \"66dd82e6-51ac-4502-8c83-6befbb446cff\") " Aug 04 16:47:41.954997 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:41.954979 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66dd82e6-51ac-4502-8c83-6befbb446cff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "66dd82e6-51ac-4502-8c83-6befbb446cff" (UID: "66dd82e6-51ac-4502-8c83-6befbb446cff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:47:42.055244 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:42.055182 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66dd82e6-51ac-4502-8c83-6befbb446cff-maas-api-service-ca\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:47:42.730566 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:42.730536 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9" Aug 04 16:47:42.730739 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:42.730533 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9" event={"ID":"66dd82e6-51ac-4502-8c83-6befbb446cff","Type":"ContainerDied","Data":"a909a7e9ab61b4ab246ee86168bf94ea7e54ce684f1cd3a5ea2087de80455d00"} Aug 04 16:47:42.730739 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:42.730658 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a909a7e9ab61b4ab246ee86168bf94ea7e54ce684f1cd3a5ea2087de80455d00" Aug 04 16:47:54.943415 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:54.943376 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8"] Aug 04 16:47:54.943981 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:54.943621 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" podUID="729a4559-9a57-4e05-9068-acce71cc9519" containerName="payload-pre-processing" containerID="cri-o://742217a6d04d0b3444573b5cbc6e12313f5492620c27303e37a17e1325e0aba5" gracePeriod=30 Aug 04 16:47:55.088965 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.088931 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9"] Aug 04 16:47:55.092148 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.092119 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2847c7-bs2v9"] Aug 04 16:47:55.190103 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.190079 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" Aug 04 16:47:55.255347 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.255257 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcpd6\" (UniqueName: \"kubernetes.io/projected/729a4559-9a57-4e05-9068-acce71cc9519-kube-api-access-gcpd6\") pod \"729a4559-9a57-4e05-9068-acce71cc9519\" (UID: \"729a4559-9a57-4e05-9068-acce71cc9519\") " Aug 04 16:47:55.255347 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.255323 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729a4559-9a57-4e05-9068-acce71cc9519-plugins-config-volume\") pod \"729a4559-9a57-4e05-9068-acce71cc9519\" (UID: \"729a4559-9a57-4e05-9068-acce71cc9519\") " Aug 04 16:47:55.255653 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.255629 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729a4559-9a57-4e05-9068-acce71cc9519-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "729a4559-9a57-4e05-9068-acce71cc9519" (UID: "729a4559-9a57-4e05-9068-acce71cc9519"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:47:55.257315 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.257271 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729a4559-9a57-4e05-9068-acce71cc9519-kube-api-access-gcpd6" (OuterVolumeSpecName: "kube-api-access-gcpd6") pod "729a4559-9a57-4e05-9068-acce71cc9519" (UID: "729a4559-9a57-4e05-9068-acce71cc9519"). InnerVolumeSpecName "kube-api-access-gcpd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:47:55.319414 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.319381 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66dd82e6-51ac-4502-8c83-6befbb446cff" path="/var/lib/kubelet/pods/66dd82e6-51ac-4502-8c83-6befbb446cff/volumes" Aug 04 16:47:55.356454 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.356420 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcpd6\" (UniqueName: \"kubernetes.io/projected/729a4559-9a57-4e05-9068-acce71cc9519-kube-api-access-gcpd6\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:47:55.356454 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.356443 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729a4559-9a57-4e05-9068-acce71cc9519-plugins-config-volume\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:47:55.798410 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.798378 2580 generic.go:356] "Generic (PLEG): container finished" podID="729a4559-9a57-4e05-9068-acce71cc9519" containerID="742217a6d04d0b3444573b5cbc6e12313f5492620c27303e37a17e1325e0aba5" exitCode=0 Aug 04 16:47:55.798581 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.798426 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" Aug 04 16:47:55.798581 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.798467 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" event={"ID":"729a4559-9a57-4e05-9068-acce71cc9519","Type":"ContainerDied","Data":"742217a6d04d0b3444573b5cbc6e12313f5492620c27303e37a17e1325e0aba5"} Aug 04 16:47:55.798581 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.798533 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8" event={"ID":"729a4559-9a57-4e05-9068-acce71cc9519","Type":"ContainerDied","Data":"ecb3c3af71ad9315256b38bbfdf2b8ecf83bae792d7a3dd7ca3c4805e4327aac"} Aug 04 16:47:55.798581 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.798556 2580 scope.go:117] "RemoveContainer" containerID="742217a6d04d0b3444573b5cbc6e12313f5492620c27303e37a17e1325e0aba5" Aug 04 16:47:55.806845 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.806825 2580 scope.go:117] "RemoveContainer" containerID="742217a6d04d0b3444573b5cbc6e12313f5492620c27303e37a17e1325e0aba5" Aug 04 16:47:55.807099 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:47:55.807078 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"742217a6d04d0b3444573b5cbc6e12313f5492620c27303e37a17e1325e0aba5\": container with ID starting with 742217a6d04d0b3444573b5cbc6e12313f5492620c27303e37a17e1325e0aba5 not found: ID does not exist" containerID="742217a6d04d0b3444573b5cbc6e12313f5492620c27303e37a17e1325e0aba5" Aug 04 16:47:55.807177 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.807111 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742217a6d04d0b3444573b5cbc6e12313f5492620c27303e37a17e1325e0aba5"} err="failed to get container status \"742217a6d04d0b3444573b5cbc6e12313f5492620c27303e37a17e1325e0aba5\": rpc error: code = NotFound desc = could not find container \"742217a6d04d0b3444573b5cbc6e12313f5492620c27303e37a17e1325e0aba5\": container with ID starting with 742217a6d04d0b3444573b5cbc6e12313f5492620c27303e37a17e1325e0aba5 not found: ID does not exist" Aug 04 16:47:55.817738 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.817719 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8"] Aug 04 16:47:55.823442 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:55.823421 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2847c7-75878bfcb4-j2ws8"] Aug 04 16:47:56.934159 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:56.934124 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:47:57.319120 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:47:57.319048 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="729a4559-9a57-4e05-9068-acce71cc9519" path="/var/lib/kubelet/pods/729a4559-9a57-4e05-9068-acce71cc9519/volumes" Aug 04 16:48:07.976198 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:07.976122 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh"] Aug 04 16:48:07.976619 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:07.976478 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="729a4559-9a57-4e05-9068-acce71cc9519" containerName="payload-pre-processing" Aug 04 16:48:07.976619 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:07.976491 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="729a4559-9a57-4e05-9068-acce71cc9519" containerName="payload-pre-processing" Aug 04 16:48:07.976619 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:07.976514 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66dd82e6-51ac-4502-8c83-6befbb446cff" containerName="revoke-keys" Aug 04 16:48:07.976619 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:07.976520 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="66dd82e6-51ac-4502-8c83-6befbb446cff" containerName="revoke-keys" Aug 04 16:48:07.976619 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:07.976570 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="729a4559-9a57-4e05-9068-acce71cc9519" containerName="payload-pre-processing" Aug 04 16:48:07.976619 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:07.976580 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="66dd82e6-51ac-4502-8c83-6befbb446cff" containerName="revoke-keys" Aug 04 16:48:07.980988 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:07.980966 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" Aug 04 16:48:07.984523 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:07.984504 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-4e1c51-dockercfg-w8nsx\"" Aug 04 16:48:07.984608 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:07.984595 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-4e1c51\"" Aug 04 16:48:07.993741 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:07.993721 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh"] Aug 04 16:48:08.012920 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.012896 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h"] Aug 04 16:48:08.016046 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.016031 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" Aug 04 16:48:08.029606 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.029583 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h"] Aug 04 16:48:08.054206 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.054184 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2371e1d3-86fc-43cd-879d-0baa283e5eac-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh\" (UID: \"2371e1d3-86fc-43cd-879d-0baa283e5eac\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" Aug 04 16:48:08.054352 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.054224 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrpx2\" (UniqueName: \"kubernetes.io/projected/2371e1d3-86fc-43cd-879d-0baa283e5eac-kube-api-access-vrpx2\") pod \"payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh\" (UID: \"2371e1d3-86fc-43cd-879d-0baa283e5eac\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" Aug 04 16:48:08.155053 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.155027 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msmpw\" (UniqueName: \"kubernetes.io/projected/f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb-kube-api-access-msmpw\") pod \"payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h\" (UID: \"f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" Aug 04 16:48:08.155197 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.155064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2371e1d3-86fc-43cd-879d-0baa283e5eac-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh\" (UID: \"2371e1d3-86fc-43cd-879d-0baa283e5eac\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" Aug 04 16:48:08.155197 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.155086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h\" (UID: \"f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" Aug 04 16:48:08.155271 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.155238 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrpx2\" (UniqueName: \"kubernetes.io/projected/2371e1d3-86fc-43cd-879d-0baa283e5eac-kube-api-access-vrpx2\") pod \"payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh\" (UID: \"2371e1d3-86fc-43cd-879d-0baa283e5eac\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" Aug 04 16:48:08.155721 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.155704 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2371e1d3-86fc-43cd-879d-0baa283e5eac-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh\" (UID: \"2371e1d3-86fc-43cd-879d-0baa283e5eac\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" Aug 04 16:48:08.163325 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.163304 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrpx2\" (UniqueName: \"kubernetes.io/projected/2371e1d3-86fc-43cd-879d-0baa283e5eac-kube-api-access-vrpx2\") pod \"payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh\" (UID: \"2371e1d3-86fc-43cd-879d-0baa283e5eac\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" Aug 04 16:48:08.255851 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.255786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msmpw\" (UniqueName: \"kubernetes.io/projected/f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb-kube-api-access-msmpw\") pod \"payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h\" (UID: \"f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" Aug 04 16:48:08.255851 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.255830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h\" (UID: \"f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" Aug 04 16:48:08.256530 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.256509 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h\" (UID: \"f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" Aug 04 16:48:08.264308 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.264271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msmpw\" (UniqueName: \"kubernetes.io/projected/f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb-kube-api-access-msmpw\") pod \"payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h\" (UID: \"f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" Aug 04 16:48:08.291761 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.291743 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" Aug 04 16:48:08.325554 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.325532 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" Aug 04 16:48:08.433643 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.433547 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh"] Aug 04 16:48:08.436469 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:48:08.436443 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2371e1d3_86fc_43cd_879d_0baa283e5eac.slice/crio-b5017ccff49319d0621f54e21a52c85eaad038a7c600a784f1fb55850bf2a8c0 WatchSource:0}: Error finding container b5017ccff49319d0621f54e21a52c85eaad038a7c600a784f1fb55850bf2a8c0: Status 404 returned error can't find the container with id b5017ccff49319d0621f54e21a52c85eaad038a7c600a784f1fb55850bf2a8c0 Aug 04 16:48:08.679224 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.679132 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h"] Aug 04 16:48:08.681205 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:48:08.681178 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a37ce6_8a0f_47b8_856e_e9cc9112f9bb.slice/crio-06af82a18b42e465a455c068c9b01aa91faedfa90db3a3dd2fb7cf718f7ce68d WatchSource:0}: Error finding container 06af82a18b42e465a455c068c9b01aa91faedfa90db3a3dd2fb7cf718f7ce68d: Status 404 returned error can't find the container with id 06af82a18b42e465a455c068c9b01aa91faedfa90db3a3dd2fb7cf718f7ce68d Aug 04 16:48:08.863571 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.863540 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" event={"ID":"f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb","Type":"ContainerStarted","Data":"3a4d7d24858da5364de5aac884c5fcba1261a5f45b91bb2086d3db4a5eed60c0"} Aug 04 16:48:08.863842 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.863642 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" Aug 04 16:48:08.863842 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.863691 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" event={"ID":"f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb","Type":"ContainerStarted","Data":"06af82a18b42e465a455c068c9b01aa91faedfa90db3a3dd2fb7cf718f7ce68d"} Aug 04 16:48:08.865166 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.865140 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" event={"ID":"2371e1d3-86fc-43cd-879d-0baa283e5eac","Type":"ContainerStarted","Data":"07e0fcbc5fb2742dc7365fe15ef57f05dcc9eefb1fffed6adc52cc10a0c145eb"} Aug 04 16:48:08.865261 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.865189 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" event={"ID":"2371e1d3-86fc-43cd-879d-0baa283e5eac","Type":"ContainerStarted","Data":"b5017ccff49319d0621f54e21a52c85eaad038a7c600a784f1fb55850bf2a8c0"} Aug 04 16:48:08.865261 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.865211 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" Aug 04 16:48:08.881034 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.880992 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" podStartSLOduration=1.880978681 podStartE2EDuration="1.880978681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:48:08.880554057 +0000 UTC m=+2790.024886366" watchObservedRunningTime="2026-08-04 16:48:08.880978681 +0000 UTC m=+2790.025310927" Aug 04 16:48:08.898268 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:08.898231 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" podStartSLOduration=1.898217837 podStartE2EDuration="1.898217837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:48:08.897830187 +0000 UTC m=+2790.042162452" watchObservedRunningTime="2026-08-04 16:48:08.898217837 +0000 UTC m=+2790.042550081" Aug 04 16:48:09.844402 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:09.844369 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:48:19.872307 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:19.872264 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" Aug 04 16:48:19.872878 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:19.872673 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" Aug 04 16:48:36.457800 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.457767 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95"] Aug 04 16:48:36.461268 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.461250 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" Aug 04 16:48:36.464443 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.464415 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-222eca-dockercfg-tkq66\"" Aug 04 16:48:36.464579 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.464443 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-222eca\"" Aug 04 16:48:36.475390 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.475338 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95"] Aug 04 16:48:36.475645 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.475615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dml2\" (UniqueName: \"kubernetes.io/projected/15434214-f271-4492-8d0e-71621239d77a-kube-api-access-8dml2\") pod \"payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95\" (UID: \"15434214-f271-4492-8d0e-71621239d77a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" Aug 04 16:48:36.475768 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.475713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15434214-f271-4492-8d0e-71621239d77a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95\" (UID: \"15434214-f271-4492-8d0e-71621239d77a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" Aug 04 16:48:36.477234 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.477214 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz"] Aug 04 16:48:36.480652 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.480636 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" Aug 04 16:48:36.497084 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.497065 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz"] Aug 04 16:48:36.576820 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.576794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w28wr\" (UniqueName: \"kubernetes.io/projected/4f753cb0-606f-4bf6-a064-c5071fc35c61-kube-api-access-w28wr\") pod \"payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz\" (UID: \"4f753cb0-606f-4bf6-a064-c5071fc35c61\") " pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" Aug 04 16:48:36.576931 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.576830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dml2\" (UniqueName: \"kubernetes.io/projected/15434214-f271-4492-8d0e-71621239d77a-kube-api-access-8dml2\") pod \"payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95\" (UID: \"15434214-f271-4492-8d0e-71621239d77a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" Aug 04 16:48:36.576931 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.576903 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15434214-f271-4492-8d0e-71621239d77a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95\" (UID: \"15434214-f271-4492-8d0e-71621239d77a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" Aug 04 16:48:36.577014 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.576945 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f753cb0-606f-4bf6-a064-c5071fc35c61-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz\" (UID: \"4f753cb0-606f-4bf6-a064-c5071fc35c61\") " pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" Aug 04 16:48:36.577682 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.577657 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15434214-f271-4492-8d0e-71621239d77a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95\" (UID: \"15434214-f271-4492-8d0e-71621239d77a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" Aug 04 16:48:36.584957 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.584933 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dml2\" (UniqueName: \"kubernetes.io/projected/15434214-f271-4492-8d0e-71621239d77a-kube-api-access-8dml2\") pod \"payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95\" (UID: \"15434214-f271-4492-8d0e-71621239d77a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" Aug 04 16:48:36.677391 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.677361 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f753cb0-606f-4bf6-a064-c5071fc35c61-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz\" (UID: \"4f753cb0-606f-4bf6-a064-c5071fc35c61\") " pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" Aug 04 16:48:36.677547 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.677420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w28wr\" (UniqueName: \"kubernetes.io/projected/4f753cb0-606f-4bf6-a064-c5071fc35c61-kube-api-access-w28wr\") pod \"payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz\" (UID: \"4f753cb0-606f-4bf6-a064-c5071fc35c61\") " pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" Aug 04 16:48:36.678070 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.678043 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f753cb0-606f-4bf6-a064-c5071fc35c61-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz\" (UID: \"4f753cb0-606f-4bf6-a064-c5071fc35c61\") " pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" Aug 04 16:48:36.686425 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.686397 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w28wr\" (UniqueName: \"kubernetes.io/projected/4f753cb0-606f-4bf6-a064-c5071fc35c61-kube-api-access-w28wr\") pod \"payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz\" (UID: \"4f753cb0-606f-4bf6-a064-c5071fc35c61\") " pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" Aug 04 16:48:36.776881 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.776816 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" Aug 04 16:48:36.790957 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:36.790935 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" Aug 04 16:48:37.119015 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:37.118986 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95"] Aug 04 16:48:37.122054 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:48:37.122020 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15434214_f271_4492_8d0e_71621239d77a.slice/crio-83548b6b7fb3793a7840e175360384badee4957f9cfa21d1bbec222fda734f18 WatchSource:0}: Error finding container 83548b6b7fb3793a7840e175360384badee4957f9cfa21d1bbec222fda734f18: Status 404 returned error can't find the container with id 83548b6b7fb3793a7840e175360384badee4957f9cfa21d1bbec222fda734f18 Aug 04 16:48:37.151132 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:37.151072 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz"] Aug 04 16:48:37.153424 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:48:37.153401 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f753cb0_606f_4bf6_a064_c5071fc35c61.slice/crio-3a05cbf7dd7e285a0227652b5105f73285e68e2885e6f99c29560913a4c200ee WatchSource:0}: Error finding container 3a05cbf7dd7e285a0227652b5105f73285e68e2885e6f99c29560913a4c200ee: Status 404 returned error can't find the container with id 3a05cbf7dd7e285a0227652b5105f73285e68e2885e6f99c29560913a4c200ee Aug 04 16:48:38.013834 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:38.013799 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" event={"ID":"4f753cb0-606f-4bf6-a064-c5071fc35c61","Type":"ContainerStarted","Data":"baaf5f94baa9f2b70f5160cb5bc80ec5b9cbbcb5c10211ceb07f8a728ec46152"} Aug 04 16:48:38.014214 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:38.013854 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" event={"ID":"4f753cb0-606f-4bf6-a064-c5071fc35c61","Type":"ContainerStarted","Data":"3a05cbf7dd7e285a0227652b5105f73285e68e2885e6f99c29560913a4c200ee"} Aug 04 16:48:38.014214 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:38.013891 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" Aug 04 16:48:38.015457 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:38.015431 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" event={"ID":"15434214-f271-4492-8d0e-71621239d77a","Type":"ContainerStarted","Data":"43eb054afbc5d61d87dffff5cd2d5cee31f65c6084808d24bd275e6e2b32a044"} Aug 04 16:48:38.015564 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:38.015482 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" Aug 04 16:48:38.015564 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:38.015508 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" event={"ID":"15434214-f271-4492-8d0e-71621239d77a","Type":"ContainerStarted","Data":"83548b6b7fb3793a7840e175360384badee4957f9cfa21d1bbec222fda734f18"} Aug 04 16:48:38.034626 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:38.034580 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" podStartSLOduration=2.034566991 podStartE2EDuration="2.034566991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:48:38.0334645 +0000 UTC m=+2819.177796772" watchObservedRunningTime="2026-08-04 16:48:38.034566991 +0000 UTC m=+2819.178899235" Aug 04 16:48:38.051647 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:38.051602 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" podStartSLOduration=2.051589873 podStartE2EDuration="2.051589873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:48:38.050709579 +0000 UTC m=+2819.195041843" watchObservedRunningTime="2026-08-04 16:48:38.051589873 +0000 UTC m=+2819.195922164" Aug 04 16:48:38.539887 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:38.539855 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:48:49.022121 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:49.022091 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" Aug 04 16:48:49.022580 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:48:49.022160 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" Aug 04 16:49:22.234938 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:49:22.234900 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:50:06.850190 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:50:06.850153 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:50:30.843271 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:50:30.843235 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:50:46.633314 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:50:46.633265 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:50:57.133909 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:50:57.133873 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:51:13.229797 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:51:13.229711 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:51:28.735839 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:51:28.735806 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:51:39.245255 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:51:39.245218 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:51:39.442659 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:51:39.442631 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:51:39.450948 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:51:39.450928 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:51:39.593055 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:51:39.593022 2580 scope.go:117] "RemoveContainer" containerID="53d270eaf05ef081dd639fa1c5084559f8906ee60170d706f7cd49fe3730bd00" Aug 04 16:51:39.603058 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:51:39.603034 2580 scope.go:117] "RemoveContainer" containerID="067af14d2dc88489b37a4af31d547055085597c4a423278d17d9ee566535ad7a" Aug 04 16:51:55.431434 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:51:55.431399 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:52:10.639669 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:52:10.639629 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:52:21.133513 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:52:21.133481 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:52:37.032108 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:52:37.032033 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:52:39.624604 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:52:39.624571 2580 scope.go:117] "RemoveContainer" containerID="8df7c503e23d718984cc9943a584e3afc2036a39ba7e324e59a92c79837876ed" Aug 04 16:52:39.632985 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:52:39.632964 2580 scope.go:117] "RemoveContainer" containerID="1a61e49b13bb7b418508c985e4f9e6618375874b046b981e00f12e0c55625c5f" Aug 04 16:52:53.336915 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:52:53.336887 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:53:04.948369 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:53:04.948341 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:53:19.730111 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:53:19.730079 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:53:30.232969 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:53:30.232936 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:53:39.654045 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:53:39.654010 2580 scope.go:117] "RemoveContainer" containerID="ee99c09b8a5f0af4ed410289eb3e1a9dd7d3646f87d211f30056eb0b18a4566c" Aug 04 16:53:45.831424 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:53:45.831392 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:53:59.429234 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:53:59.429203 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:54:08.928112 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:08.928039 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c"] Aug 04 16:54:08.931632 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:08.931615 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:08.934587 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:08.934558 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4e1c51\"/\"openshift-service-ca.crt\"" Aug 04 16:54:08.934712 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:08.934589 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4e1c51\"/\"kube-root-ca.crt\"" Aug 04 16:54:08.934712 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:08.934566 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4e1c51\"/\"default-dockercfg-s64fs\"" Aug 04 16:54:08.935552 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:08.935533 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4e1c51\"/\"sub-test-model-4e1c51-kserve-self-signed-certs\"" Aug 04 16:54:08.941146 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:08.941123 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c"] Aug 04 16:54:09.098338 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.098308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-home\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.098485 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.098361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4gnc\" (UniqueName: \"kubernetes.io/projected/026e176f-3557-4869-9c5c-88d7164bccb2-kube-api-access-k4gnc\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.098485 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.098432 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-dshm\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.098485 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.098463 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-model-cache\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.098640 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.098491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-tmp-dir\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.098640 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.098555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026e176f-3557-4869-9c5c-88d7164bccb2-tls-certs\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.199562 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.199489 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-home\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.199562 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.199542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4gnc\" (UniqueName: \"kubernetes.io/projected/026e176f-3557-4869-9c5c-88d7164bccb2-kube-api-access-k4gnc\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.199746 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.199614 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-dshm\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.199746 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.199637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-model-cache\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.199746 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.199665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-tmp-dir\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.199746 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.199696 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026e176f-3557-4869-9c5c-88d7164bccb2-tls-certs\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.199952 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.199856 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-home\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.200024 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.199997 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-model-cache\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.200087 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.200002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-tmp-dir\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.201779 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.201759 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-dshm\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.202055 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.202037 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026e176f-3557-4869-9c5c-88d7164bccb2-tls-certs\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.207773 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.207749 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4gnc\" (UniqueName: \"kubernetes.io/projected/026e176f-3557-4869-9c5c-88d7164bccb2-kube-api-access-k4gnc\") pod \"sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.241031 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.241009 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:09.380156 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.380132 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c"] Aug 04 16:54:09.382198 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:54:09.382172 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod026e176f_3557_4869_9c5c_88d7164bccb2.slice/crio-fc045a5e09c8f99df376f9284ef47c91026965b124af01b324b880cdf7f73996 WatchSource:0}: Error finding container fc045a5e09c8f99df376f9284ef47c91026965b124af01b324b880cdf7f73996: Status 404 returned error can't find the container with id fc045a5e09c8f99df376f9284ef47c91026965b124af01b324b880cdf7f73996 Aug 04 16:54:09.610757 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.610722 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" event={"ID":"026e176f-3557-4869-9c5c-88d7164bccb2","Type":"ContainerStarted","Data":"fd4990514807b6ad807731d05559d8825dfc0656ef7a2be69c9f6cfb663c35da"} Aug 04 16:54:09.610909 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.610770 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" event={"ID":"026e176f-3557-4869-9c5c-88d7164bccb2","Type":"ContainerStarted","Data":"fc045a5e09c8f99df376f9284ef47c91026965b124af01b324b880cdf7f73996"} Aug 04 16:54:09.628259 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:09.628171 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" podStartSLOduration=1.6281551269999999 podStartE2EDuration="1.628155127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:54:09.627981779 +0000 UTC m=+3150.772314034" watchObservedRunningTime="2026-08-04 16:54:09.628155127 +0000 UTC m=+3150.772487374" Aug 04 16:54:10.731186 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:10.731152 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:54:19.241380 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:19.241348 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:19.241751 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:19.241405 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:19.253122 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:19.253101 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:19.671896 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:19.671872 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:54:39.675188 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:39.675158 2580 scope.go:117] "RemoveContainer" containerID="18cfbd386280b4c5c56fdbc2c0a9b6a8fed485c50eb51e9c81545dda64cbd239" Aug 04 16:54:46.238203 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:54:46.238167 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:55:21.835701 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:55:21.835662 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:55:34.440360 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:55:34.440263 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:55:46.331290 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:55:46.331246 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:55:50.633381 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:55:50.633352 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c"] Aug 04 16:55:57.738148 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:55:57.738113 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:56:05.653635 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.653598 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" podUID="026e176f-3557-4869-9c5c-88d7164bccb2" containerName="main" containerID="cri-o://fd4990514807b6ad807731d05559d8825dfc0656ef7a2be69c9f6cfb663c35da" gracePeriod=45 Aug 04 16:56:05.892899 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.892876 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:56:05.993814 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.993758 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-home\") pod \"026e176f-3557-4869-9c5c-88d7164bccb2\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " Aug 04 16:56:05.993814 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.993805 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-tmp-dir\") pod \"026e176f-3557-4869-9c5c-88d7164bccb2\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " Aug 04 16:56:05.993981 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.993836 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026e176f-3557-4869-9c5c-88d7164bccb2-tls-certs\") pod \"026e176f-3557-4869-9c5c-88d7164bccb2\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " Aug 04 16:56:05.993981 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.993861 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-dshm\") pod \"026e176f-3557-4869-9c5c-88d7164bccb2\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " Aug 04 16:56:05.993981 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.993897 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-model-cache\") pod \"026e176f-3557-4869-9c5c-88d7164bccb2\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " Aug 04 16:56:05.994144 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.994017 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-home" (OuterVolumeSpecName: "home") pod "026e176f-3557-4869-9c5c-88d7164bccb2" (UID: "026e176f-3557-4869-9c5c-88d7164bccb2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:56:05.994144 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.994069 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4gnc\" (UniqueName: \"kubernetes.io/projected/026e176f-3557-4869-9c5c-88d7164bccb2-kube-api-access-k4gnc\") pod \"026e176f-3557-4869-9c5c-88d7164bccb2\" (UID: \"026e176f-3557-4869-9c5c-88d7164bccb2\") " Aug 04 16:56:05.994144 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.994118 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "026e176f-3557-4869-9c5c-88d7164bccb2" (UID: "026e176f-3557-4869-9c5c-88d7164bccb2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:56:05.994312 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.994129 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-model-cache" (OuterVolumeSpecName: "model-cache") pod "026e176f-3557-4869-9c5c-88d7164bccb2" (UID: "026e176f-3557-4869-9c5c-88d7164bccb2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:56:05.994369 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.994352 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-model-cache\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:56:05.994410 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.994375 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-home\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:56:05.994410 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.994388 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-tmp-dir\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:56:05.995973 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.995952 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026e176f-3557-4869-9c5c-88d7164bccb2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "026e176f-3557-4869-9c5c-88d7164bccb2" (UID: "026e176f-3557-4869-9c5c-88d7164bccb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:56:05.995973 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.995957 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-dshm" (OuterVolumeSpecName: "dshm") pod "026e176f-3557-4869-9c5c-88d7164bccb2" (UID: "026e176f-3557-4869-9c5c-88d7164bccb2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:56:05.996120 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:05.996100 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026e176f-3557-4869-9c5c-88d7164bccb2-kube-api-access-k4gnc" (OuterVolumeSpecName: "kube-api-access-k4gnc") pod "026e176f-3557-4869-9c5c-88d7164bccb2" (UID: "026e176f-3557-4869-9c5c-88d7164bccb2"). InnerVolumeSpecName "kube-api-access-k4gnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:56:06.094992 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:06.094968 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026e176f-3557-4869-9c5c-88d7164bccb2-tls-certs\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:56:06.094992 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:06.094989 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026e176f-3557-4869-9c5c-88d7164bccb2-dshm\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:56:06.095120 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:06.094999 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4gnc\" (UniqueName: \"kubernetes.io/projected/026e176f-3557-4869-9c5c-88d7164bccb2-kube-api-access-k4gnc\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:56:06.172064 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:06.172035 2580 generic.go:356] "Generic (PLEG): container finished" podID="026e176f-3557-4869-9c5c-88d7164bccb2" containerID="fd4990514807b6ad807731d05559d8825dfc0656ef7a2be69c9f6cfb663c35da" exitCode=0 Aug 04 16:56:06.172164 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:06.172079 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" Aug 04 16:56:06.172164 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:06.172125 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" event={"ID":"026e176f-3557-4869-9c5c-88d7164bccb2","Type":"ContainerDied","Data":"fd4990514807b6ad807731d05559d8825dfc0656ef7a2be69c9f6cfb663c35da"} Aug 04 16:56:06.172302 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:06.172171 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c" event={"ID":"026e176f-3557-4869-9c5c-88d7164bccb2","Type":"ContainerDied","Data":"fc045a5e09c8f99df376f9284ef47c91026965b124af01b324b880cdf7f73996"} Aug 04 16:56:06.172302 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:06.172194 2580 scope.go:117] "RemoveContainer" containerID="fd4990514807b6ad807731d05559d8825dfc0656ef7a2be69c9f6cfb663c35da" Aug 04 16:56:06.181441 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:06.181426 2580 scope.go:117] "RemoveContainer" containerID="fd4990514807b6ad807731d05559d8825dfc0656ef7a2be69c9f6cfb663c35da" Aug 04 16:56:06.181689 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:56:06.181669 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd4990514807b6ad807731d05559d8825dfc0656ef7a2be69c9f6cfb663c35da\": container with ID starting with fd4990514807b6ad807731d05559d8825dfc0656ef7a2be69c9f6cfb663c35da not found: ID does not exist" containerID="fd4990514807b6ad807731d05559d8825dfc0656ef7a2be69c9f6cfb663c35da" Aug 04 16:56:06.181750 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:06.181700 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd4990514807b6ad807731d05559d8825dfc0656ef7a2be69c9f6cfb663c35da"} err="failed to get container status \"fd4990514807b6ad807731d05559d8825dfc0656ef7a2be69c9f6cfb663c35da\": rpc error: code = NotFound desc = could not find container \"fd4990514807b6ad807731d05559d8825dfc0656ef7a2be69c9f6cfb663c35da\": container with ID starting with fd4990514807b6ad807731d05559d8825dfc0656ef7a2be69c9f6cfb663c35da not found: ID does not exist" Aug 04 16:56:06.193166 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:06.193145 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c"] Aug 04 16:56:06.196790 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:06.196771 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4e1c51/sub-test-model-4e1c51-kserve-7c94779f7c-z8x5c"] Aug 04 16:56:07.318794 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:07.318757 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="026e176f-3557-4869-9c5c-88d7164bccb2" path="/var/lib/kubelet/pods/026e176f-3557-4869-9c5c-88d7164bccb2/volumes" Aug 04 16:56:15.731830 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:15.731790 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:56:39.473690 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:39.473664 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:56:39.480538 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:39.480516 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 16:56:44.940812 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:56:44.940778 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:57:34.640263 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:57:34.640226 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:57:48.004567 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:57:48.004534 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:57:58.840570 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:57:58.840537 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:58:06.774379 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:06.774349 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7667-140255613663920"] Aug 04 16:58:06.774845 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:06.774700 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="026e176f-3557-4869-9c5c-88d7164bccb2" containerName="main" Aug 04 16:58:06.774845 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:06.774713 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="026e176f-3557-4869-9c5c-88d7164bccb2" containerName="main" Aug 04 16:58:06.774845 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:06.774804 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="026e176f-3557-4869-9c5c-88d7164bccb2" containerName="main" Aug 04 16:58:06.777970 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:06.777952 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7667-140255613663920" Aug 04 16:58:06.780620 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:06.780600 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zj6q8\"" Aug 04 16:58:06.784574 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:06.784554 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7667-140255613663920"] Aug 04 16:58:06.902734 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:06.902713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns6gv\" (UniqueName: \"kubernetes.io/projected/d3bc5f2a-1c98-4dd6-89d4-605ba363c599-kube-api-access-ns6gv\") pod \"test-curl-7667-140255613663920\" (UID: \"d3bc5f2a-1c98-4dd6-89d4-605ba363c599\") " pod="opendatahub/test-curl-7667-140255613663920" Aug 04 16:58:07.003528 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:07.003504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns6gv\" (UniqueName: \"kubernetes.io/projected/d3bc5f2a-1c98-4dd6-89d4-605ba363c599-kube-api-access-ns6gv\") pod \"test-curl-7667-140255613663920\" (UID: \"d3bc5f2a-1c98-4dd6-89d4-605ba363c599\") " pod="opendatahub/test-curl-7667-140255613663920" Aug 04 16:58:07.012519 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:07.012491 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns6gv\" (UniqueName: \"kubernetes.io/projected/d3bc5f2a-1c98-4dd6-89d4-605ba363c599-kube-api-access-ns6gv\") pod \"test-curl-7667-140255613663920\" (UID: \"d3bc5f2a-1c98-4dd6-89d4-605ba363c599\") " pod="opendatahub/test-curl-7667-140255613663920" Aug 04 16:58:07.087657 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:07.087637 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7667-140255613663920" Aug 04 16:58:07.218073 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:07.217997 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7667-140255613663920"] Aug 04 16:58:07.220166 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:58:07.220145 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3bc5f2a_1c98_4dd6_89d4_605ba363c599.slice/crio-d9b2d628df5e167839d0564e0a29b24eae1e372e503046269742f83895562c04 WatchSource:0}: Error finding container d9b2d628df5e167839d0564e0a29b24eae1e372e503046269742f83895562c04: Status 404 returned error can't find the container with id d9b2d628df5e167839d0564e0a29b24eae1e372e503046269742f83895562c04 Aug 04 16:58:07.221481 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:07.221463 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:58:07.728080 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:07.728040 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7667-140255613663920" event={"ID":"d3bc5f2a-1c98-4dd6-89d4-605ba363c599","Type":"ContainerStarted","Data":"d9b2d628df5e167839d0564e0a29b24eae1e372e503046269742f83895562c04"} Aug 04 16:58:08.731960 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:08.731925 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:58:08.735601 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:08.735575 2580 generic.go:356] "Generic (PLEG): container finished" podID="d3bc5f2a-1c98-4dd6-89d4-605ba363c599" containerID="90f694adc78e7162a765039c469470f469e23ec7c4b08254afdcd1d110ebdf3e" exitCode=0 Aug 04 16:58:08.735717 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:08.735660 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7667-140255613663920" event={"ID":"d3bc5f2a-1c98-4dd6-89d4-605ba363c599","Type":"ContainerDied","Data":"90f694adc78e7162a765039c469470f469e23ec7c4b08254afdcd1d110ebdf3e"} Aug 04 16:58:09.876455 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:09.876432 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7667-140255613663920" Aug 04 16:58:09.913040 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:09.913011 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7667-140255613663920_d3bc5f2a-1c98-4dd6-89d4-605ba363c599/test-curl-7667-140255613663920/0.log" Aug 04 16:58:09.938853 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:09.938823 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7667-140255613663920"] Aug 04 16:58:09.942811 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:09.942793 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7667-140255613663920"] Aug 04 16:58:10.027327 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.027258 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns6gv\" (UniqueName: \"kubernetes.io/projected/d3bc5f2a-1c98-4dd6-89d4-605ba363c599-kube-api-access-ns6gv\") pod \"d3bc5f2a-1c98-4dd6-89d4-605ba363c599\" (UID: \"d3bc5f2a-1c98-4dd6-89d4-605ba363c599\") " Aug 04 16:58:10.029273 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.029244 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3bc5f2a-1c98-4dd6-89d4-605ba363c599-kube-api-access-ns6gv" (OuterVolumeSpecName: "kube-api-access-ns6gv") pod "d3bc5f2a-1c98-4dd6-89d4-605ba363c599" (UID: "d3bc5f2a-1c98-4dd6-89d4-605ba363c599"). InnerVolumeSpecName "kube-api-access-ns6gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:58:10.075862 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.075837 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7667-140255613694256"] Aug 04 16:58:10.076296 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.076267 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3bc5f2a-1c98-4dd6-89d4-605ba363c599" containerName="test-curl-7667-140255613663920" Aug 04 16:58:10.076296 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.076295 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3bc5f2a-1c98-4dd6-89d4-605ba363c599" containerName="test-curl-7667-140255613663920" Aug 04 16:58:10.076416 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.076378 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3bc5f2a-1c98-4dd6-89d4-605ba363c599" containerName="test-curl-7667-140255613663920" Aug 04 16:58:10.079387 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.079369 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7667-140255613694256" Aug 04 16:58:10.086067 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.085938 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7667-140255613694256"] Aug 04 16:58:10.128616 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.128594 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns6gv\" (UniqueName: \"kubernetes.io/projected/d3bc5f2a-1c98-4dd6-89d4-605ba363c599-kube-api-access-ns6gv\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:58:10.229905 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.229880 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r7tw\" (UniqueName: \"kubernetes.io/projected/61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4-kube-api-access-7r7tw\") pod \"test-curl-7667-140255613694256\" (UID: \"61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4\") " pod="opendatahub/test-curl-7667-140255613694256" Aug 04 16:58:10.330986 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.330964 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r7tw\" (UniqueName: \"kubernetes.io/projected/61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4-kube-api-access-7r7tw\") pod \"test-curl-7667-140255613694256\" (UID: \"61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4\") " pod="opendatahub/test-curl-7667-140255613694256" Aug 04 16:58:10.338357 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.338332 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r7tw\" (UniqueName: \"kubernetes.io/projected/61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4-kube-api-access-7r7tw\") pod \"test-curl-7667-140255613694256\" (UID: \"61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4\") " pod="opendatahub/test-curl-7667-140255613694256" Aug 04 16:58:10.388421 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.388401 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7667-140255613694256" Aug 04 16:58:10.514563 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.514536 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7667-140255613694256"] Aug 04 16:58:10.516771 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:58:10.516745 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61aa9918_e1c0_4d3b_a0ea_5531bc9be2b4.slice/crio-4e425015f7b5acf213ea2b54fc155b6b9b6bd95486496144375879321530c623 WatchSource:0}: Error finding container 4e425015f7b5acf213ea2b54fc155b6b9b6bd95486496144375879321530c623: Status 404 returned error can't find the container with id 4e425015f7b5acf213ea2b54fc155b6b9b6bd95486496144375879321530c623 Aug 04 16:58:10.746043 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.745974 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7667-140255613694256" event={"ID":"61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4","Type":"ContainerStarted","Data":"4e425015f7b5acf213ea2b54fc155b6b9b6bd95486496144375879321530c623"} Aug 04 16:58:10.747333 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.747314 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7667-140255613663920" Aug 04 16:58:10.747430 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:10.747315 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9b2d628df5e167839d0564e0a29b24eae1e372e503046269742f83895562c04" Aug 04 16:58:10.757545 ip-10-0-132-42 kubenswrapper[2580]: E0804 16:58:10.757519 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7667-140255613663920\" is forbidden: User \"system:node:ip-10-0-132-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-42.ec2.internal' and this object" podUID="d3bc5f2a-1c98-4dd6-89d4-605ba363c599" pod="opendatahub/test-curl-7667-140255613663920" Aug 04 16:58:11.319780 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:11.319752 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3bc5f2a-1c98-4dd6-89d4-605ba363c599" path="/var/lib/kubelet/pods/d3bc5f2a-1c98-4dd6-89d4-605ba363c599/volumes" Aug 04 16:58:11.753672 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:11.753640 2580 generic.go:356] "Generic (PLEG): container finished" podID="61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4" containerID="d36476e409a17865608b251c81c7529e232a7bf5f5f80c5797f25c21263e195c" exitCode=0 Aug 04 16:58:11.753830 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:11.753692 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7667-140255613694256" event={"ID":"61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4","Type":"ContainerDied","Data":"d36476e409a17865608b251c81c7529e232a7bf5f5f80c5797f25c21263e195c"} Aug 04 16:58:12.912123 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:12.912098 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7667-140255613694256" Aug 04 16:58:12.932177 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:12.932155 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7667-140255613694256_61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4/test-curl-7667-140255613694256/0.log" Aug 04 16:58:12.957197 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:12.957173 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7667-140255613694256"] Aug 04 16:58:12.958841 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:12.958821 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7667-140255613694256"] Aug 04 16:58:13.053588 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:13.053537 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r7tw\" (UniqueName: \"kubernetes.io/projected/61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4-kube-api-access-7r7tw\") pod \"61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4\" (UID: \"61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4\") " Aug 04 16:58:13.055447 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:13.055421 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4-kube-api-access-7r7tw" (OuterVolumeSpecName: "kube-api-access-7r7tw") pod "61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4" (UID: "61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4"). InnerVolumeSpecName "kube-api-access-7r7tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:58:13.154679 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:13.154658 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r7tw\" (UniqueName: \"kubernetes.io/projected/61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4-kube-api-access-7r7tw\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 16:58:13.319697 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:13.319645 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4" path="/var/lib/kubelet/pods/61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4/volumes" Aug 04 16:58:13.764701 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:13.764676 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7667-140255613694256" Aug 04 16:58:13.764858 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:13.764730 2580 scope.go:117] "RemoveContainer" containerID="d36476e409a17865608b251c81c7529e232a7bf5f5f80c5797f25c21263e195c" Aug 04 16:58:31.546902 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:31.546813 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p"] Aug 04 16:58:31.547376 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:31.547359 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4" containerName="test-curl-7667-140255613694256" Aug 04 16:58:31.547430 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:31.547378 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4" containerName="test-curl-7667-140255613694256" Aug 04 16:58:31.547472 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:31.547452 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="61aa9918-e1c0-4d3b-a0ea-5531bc9be2b4" containerName="test-curl-7667-140255613694256" Aug 04 16:58:31.550230 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:31.550213 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" Aug 04 16:58:31.553093 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:31.553067 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a125dd\"" Aug 04 16:58:31.559596 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:31.559569 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p"] Aug 04 16:58:31.597795 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:31.597768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgm9t\" (UniqueName: \"kubernetes.io/projected/79e31aaf-afb9-4aff-8052-c7f834e1832c-kube-api-access-xgm9t\") pod \"payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p\" (UID: \"79e31aaf-afb9-4aff-8052-c7f834e1832c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" Aug 04 16:58:31.597911 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:31.597810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e31aaf-afb9-4aff-8052-c7f834e1832c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p\" (UID: \"79e31aaf-afb9-4aff-8052-c7f834e1832c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" Aug 04 16:58:31.699130 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:31.699097 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgm9t\" (UniqueName: \"kubernetes.io/projected/79e31aaf-afb9-4aff-8052-c7f834e1832c-kube-api-access-xgm9t\") pod \"payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p\" (UID: \"79e31aaf-afb9-4aff-8052-c7f834e1832c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" Aug 04 16:58:31.699291 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:31.699146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e31aaf-afb9-4aff-8052-c7f834e1832c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p\" (UID: \"79e31aaf-afb9-4aff-8052-c7f834e1832c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" Aug 04 16:58:31.699791 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:31.699771 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e31aaf-afb9-4aff-8052-c7f834e1832c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p\" (UID: \"79e31aaf-afb9-4aff-8052-c7f834e1832c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" Aug 04 16:58:31.708149 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:31.708128 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgm9t\" (UniqueName: \"kubernetes.io/projected/79e31aaf-afb9-4aff-8052-c7f834e1832c-kube-api-access-xgm9t\") pod \"payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p\" (UID: \"79e31aaf-afb9-4aff-8052-c7f834e1832c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" Aug 04 16:58:31.861683 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:31.861653 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" Aug 04 16:58:32.197001 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:32.196974 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p"] Aug 04 16:58:32.199516 ip-10-0-132-42 kubenswrapper[2580]: W0804 16:58:32.199489 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e31aaf_afb9_4aff_8052_c7f834e1832c.slice/crio-a205019acb80e4ed22ae5aa441218d4f816b1329cd3b946f2efcc08ca69efba7 WatchSource:0}: Error finding container a205019acb80e4ed22ae5aa441218d4f816b1329cd3b946f2efcc08ca69efba7: Status 404 returned error can't find the container with id a205019acb80e4ed22ae5aa441218d4f816b1329cd3b946f2efcc08ca69efba7 Aug 04 16:58:32.862682 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:32.862644 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" event={"ID":"79e31aaf-afb9-4aff-8052-c7f834e1832c","Type":"ContainerStarted","Data":"c5cdde52a577e0c346024b8e61cd32daa156e8f31aed0871d462c84609fb65be"} Aug 04 16:58:32.863065 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:32.862709 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" event={"ID":"79e31aaf-afb9-4aff-8052-c7f834e1832c","Type":"ContainerStarted","Data":"a205019acb80e4ed22ae5aa441218d4f816b1329cd3b946f2efcc08ca69efba7"} Aug 04 16:58:32.863065 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:32.862899 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" Aug 04 16:58:32.880836 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:32.880794 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" podStartSLOduration=1.8807806070000002 podStartE2EDuration="1.880780607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:58:32.880156841 +0000 UTC m=+3414.024489109" watchObservedRunningTime="2026-08-04 16:58:32.880780607 +0000 UTC m=+3414.025112852" Aug 04 16:58:33.933951 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:33.933917 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:58:43.869298 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:43.869258 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" Aug 04 16:58:59.958917 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:59.958884 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs"] Aug 04 16:58:59.962217 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:59.962196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" Aug 04 16:58:59.968260 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:59.968235 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3589b4-dockercfg-2vfjl\"" Aug 04 16:58:59.969596 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:59.969575 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3589b4\"" Aug 04 16:58:59.971668 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:59.971645 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb"] Aug 04 16:58:59.974839 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:59.974821 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" Aug 04 16:58:59.976954 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:59.976934 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs"] Aug 04 16:58:59.989130 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:58:59.989108 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb"] Aug 04 16:59:00.119351 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:00.119324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44td7\" (UniqueName: \"kubernetes.io/projected/477d5ef1-a101-4dec-b30c-aeba3bb607ae-kube-api-access-44td7\") pod \"payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs\" (UID: \"477d5ef1-a101-4dec-b30c-aeba3bb607ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" Aug 04 16:59:00.119488 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:00.119354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ae85abe-7737-48f6-abea-f9445ace7164-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb\" (UID: \"2ae85abe-7737-48f6-abea-f9445ace7164\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" Aug 04 16:59:00.119488 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:00.119441 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477d5ef1-a101-4dec-b30c-aeba3bb607ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs\" (UID: \"477d5ef1-a101-4dec-b30c-aeba3bb607ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" Aug 04 16:59:00.119564 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:00.119500 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czdd2\" (UniqueName: \"kubernetes.io/projected/2ae85abe-7737-48f6-abea-f9445ace7164-kube-api-access-czdd2\") pod \"payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb\" (UID: \"2ae85abe-7737-48f6-abea-f9445ace7164\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" Aug 04 16:59:00.220270 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:00.220206 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44td7\" (UniqueName: \"kubernetes.io/projected/477d5ef1-a101-4dec-b30c-aeba3bb607ae-kube-api-access-44td7\") pod \"payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs\" (UID: \"477d5ef1-a101-4dec-b30c-aeba3bb607ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" Aug 04 16:59:00.220270 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:00.220235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ae85abe-7737-48f6-abea-f9445ace7164-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb\" (UID: \"2ae85abe-7737-48f6-abea-f9445ace7164\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" Aug 04 16:59:00.220476 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:00.220300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477d5ef1-a101-4dec-b30c-aeba3bb607ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs\" (UID: \"477d5ef1-a101-4dec-b30c-aeba3bb607ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" Aug 04 16:59:00.220476 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:00.220326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czdd2\" (UniqueName: \"kubernetes.io/projected/2ae85abe-7737-48f6-abea-f9445ace7164-kube-api-access-czdd2\") pod \"payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb\" (UID: \"2ae85abe-7737-48f6-abea-f9445ace7164\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" Aug 04 16:59:00.221052 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:00.221002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477d5ef1-a101-4dec-b30c-aeba3bb607ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs\" (UID: \"477d5ef1-a101-4dec-b30c-aeba3bb607ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" Aug 04 16:59:00.221052 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:00.221033 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ae85abe-7737-48f6-abea-f9445ace7164-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb\" (UID: \"2ae85abe-7737-48f6-abea-f9445ace7164\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" Aug 04 16:59:00.228174 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:00.228153 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czdd2\" (UniqueName: \"kubernetes.io/projected/2ae85abe-7737-48f6-abea-f9445ace7164-kube-api-access-czdd2\") pod \"payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb\" (UID: \"2ae85abe-7737-48f6-abea-f9445ace7164\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" Aug 04 16:59:00.228271 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:00.228191 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44td7\" (UniqueName: \"kubernetes.io/projected/477d5ef1-a101-4dec-b30c-aeba3bb607ae-kube-api-access-44td7\") pod \"payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs\" (UID: \"477d5ef1-a101-4dec-b30c-aeba3bb607ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" Aug 04 16:59:00.272155 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:00.272136 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" Aug 04 16:59:00.286360 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:00.286292 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" Aug 04 16:59:00.428333 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:00.426852 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs"] Aug 04 16:59:00.658147 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:00.658120 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb"] Aug 04 16:59:01.005792 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:01.005749 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" event={"ID":"477d5ef1-a101-4dec-b30c-aeba3bb607ae","Type":"ContainerStarted","Data":"826ec4e23c9aa40c5407a564dba8ad1682a544898b2b75b2a792532a96b193c6"} Aug 04 16:59:01.006222 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:01.005805 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" event={"ID":"477d5ef1-a101-4dec-b30c-aeba3bb607ae","Type":"ContainerStarted","Data":"79e53667c01bb20b020f464db47f6e1439550fb94bef1945b8158f903ab04300"} Aug 04 16:59:01.006222 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:01.005915 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" Aug 04 16:59:01.007180 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:01.007160 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" event={"ID":"2ae85abe-7737-48f6-abea-f9445ace7164","Type":"ContainerStarted","Data":"77055dfdae03752bdb883d91f40d101af8a29dcc3080585523650838d4469974"} Aug 04 16:59:01.007236 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:01.007201 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" event={"ID":"2ae85abe-7737-48f6-abea-f9445ace7164","Type":"ContainerStarted","Data":"c3777a95df0703e5312b16156be7620883b97651c4d85fd7e4f9283d730b0d38"} Aug 04 16:59:01.007323 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:01.007308 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" Aug 04 16:59:01.022891 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:01.022833 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" podStartSLOduration=2.022820562 podStartE2EDuration="2.022820562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:59:01.021249354 +0000 UTC m=+3442.165581628" watchObservedRunningTime="2026-08-04 16:59:01.022820562 +0000 UTC m=+3442.167152807" Aug 04 16:59:01.036705 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:01.036665 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" podStartSLOduration=2.036653182 podStartE2EDuration="2.036653182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:59:01.036066572 +0000 UTC m=+3442.180398844" watchObservedRunningTime="2026-08-04 16:59:01.036653182 +0000 UTC m=+3442.180985427" Aug 04 16:59:03.032884 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:03.032841 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:59:12.013032 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:12.012998 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" Aug 04 16:59:12.013526 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:12.013342 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" Aug 04 16:59:19.920789 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:19.920755 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb_2ae85abe-7737-48f6-abea-f9445ace7164/payload-processing/0.log" Aug 04 16:59:29.636181 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:29.636144 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 16:59:59.031394 ip-10-0-132-42 kubenswrapper[2580]: I0804 16:59:59.031359 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 17:00:00.134481 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:00.134453 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a125dd-29764380-sgtb7"] Aug 04 17:00:00.138159 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:00.138135 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3589b4-29764380-9ttlw"] Aug 04 17:00:00.138328 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:00.138300 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a125dd-29764380-sgtb7" Aug 04 17:00:00.140836 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:00.140817 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4pxfk\"" Aug 04 17:00:00.141906 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:00.141890 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764380-j2kbr"] Aug 04 17:00:00.142019 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:00.142004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3589b4-29764380-9ttlw" Aug 04 17:00:00.145024 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:00.145008 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764380-j2kbr" Aug 04 17:00:00.150637 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:00.150435 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a125dd-29764380-sgtb7" Aug 04 17:00:00.150898 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:00.150831 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a125dd-29764380-sgtb7"] Aug 04 17:00:00.155009 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:00.153636 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3589b4-29764380-9ttlw"] Aug 04 17:00:00.159018 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:00.156354 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764380-j2kbr"] Aug 04 17:00:00.164363 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:00.164339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3589b4-29764380-9ttlw" Aug 04 17:00:00.201066 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:00.201043 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764380-j2kbr" Aug 04 17:00:00.520349 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:00.520327 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a125dd-29764380-sgtb7"] Aug 04 17:00:00.522836 ip-10-0-132-42 kubenswrapper[2580]: W0804 17:00:00.522806 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0288e786_b008_4b87_9598_1c1ed3644f6b.slice/crio-bfbe9bc7875606a6989523a01edd49a909ab574e8855ee7d21961c3479b50a23 WatchSource:0}: Error finding container bfbe9bc7875606a6989523a01edd49a909ab574e8855ee7d21961c3479b50a23: Status 404 returned error can't find the container with id bfbe9bc7875606a6989523a01edd49a909ab574e8855ee7d21961c3479b50a23 Aug 04 17:00:00.547144 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:00.547118 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3589b4-29764380-9ttlw"] Aug 04 17:00:00.552450 ip-10-0-132-42 kubenswrapper[2580]: W0804 17:00:00.552426 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7d3634_aa65_497a_909c_b0f048531805.slice/crio-415a92f0347c86680acd0897ee687324a48838584a5fc327ac0b0ddfae2ff5aa WatchSource:0}: Error finding container 415a92f0347c86680acd0897ee687324a48838584a5fc327ac0b0ddfae2ff5aa: Status 404 returned error can't find the container with id 415a92f0347c86680acd0897ee687324a48838584a5fc327ac0b0ddfae2ff5aa Aug 04 17:00:00.574552 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:00.574527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764380-j2kbr"] Aug 04 17:00:00.586391 ip-10-0-132-42 kubenswrapper[2580]: W0804 17:00:00.586366 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad0e5fc_930e_478a_abb7_f2e4b2377239.slice/crio-b11ba7c3ebce3e9284e9019b1d60b99369d155cac34c0275dd68643a4effe7c9 WatchSource:0}: Error finding container b11ba7c3ebce3e9284e9019b1d60b99369d155cac34c0275dd68643a4effe7c9: Status 404 returned error can't find the container with id b11ba7c3ebce3e9284e9019b1d60b99369d155cac34c0275dd68643a4effe7c9 Aug 04 17:00:01.320631 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:01.320550 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a125dd-29764380-sgtb7" event={"ID":"0288e786-b008-4b87-9598-1c1ed3644f6b","Type":"ContainerStarted","Data":"4637198c72289680dc1d144959b572e035d444111549607083d076f67d085c97"} Aug 04 17:00:01.320631 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:01.320605 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a125dd-29764380-sgtb7" event={"ID":"0288e786-b008-4b87-9598-1c1ed3644f6b","Type":"ContainerStarted","Data":"bfbe9bc7875606a6989523a01edd49a909ab574e8855ee7d21961c3479b50a23"} Aug 04 17:00:01.320631 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:01.320622 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3589b4-29764380-9ttlw" event={"ID":"6b7d3634-aa65-497a-909c-b0f048531805","Type":"ContainerStarted","Data":"9c0d751bbee17d0ea14bef33744f1a39db404cb22795075dc53a970c7b67964a"} Aug 04 17:00:01.321038 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:01.320637 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3589b4-29764380-9ttlw" event={"ID":"6b7d3634-aa65-497a-909c-b0f048531805","Type":"ContainerStarted","Data":"415a92f0347c86680acd0897ee687324a48838584a5fc327ac0b0ddfae2ff5aa"} Aug 04 17:00:01.321038 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:01.320651 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764380-j2kbr" event={"ID":"5ad0e5fc-930e-478a-abb7-f2e4b2377239","Type":"ContainerStarted","Data":"abf7e7186f26da622c8c928065c69441fdd9c7a65d97189277526e4741315fe5"} Aug 04 17:00:01.321038 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:01.320666 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764380-j2kbr" event={"ID":"5ad0e5fc-930e-478a-abb7-f2e4b2377239","Type":"ContainerStarted","Data":"b11ba7c3ebce3e9284e9019b1d60b99369d155cac34c0275dd68643a4effe7c9"} Aug 04 17:00:01.328923 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:01.328883 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a125dd-29764380-sgtb7" podStartSLOduration=1.3288679939999999 podStartE2EDuration="1.328867994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:00:01.327204945 +0000 UTC m=+3502.471537201" watchObservedRunningTime="2026-08-04 17:00:01.328867994 +0000 UTC m=+3502.473200239" Aug 04 17:00:01.342801 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:01.342759 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764380-j2kbr" podStartSLOduration=1.34274592 podStartE2EDuration="1.34274592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:00:01.340885351 +0000 UTC m=+3502.485217610" watchObservedRunningTime="2026-08-04 17:00:01.34274592 +0000 UTC m=+3502.487078165" Aug 04 17:00:01.355403 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:01.355368 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3589b4-29764380-9ttlw" podStartSLOduration=1.355358392 podStartE2EDuration="1.355358392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:00:01.354648828 +0000 UTC m=+3502.498981090" watchObservedRunningTime="2026-08-04 17:00:01.355358392 +0000 UTC m=+3502.499690637" Aug 04 17:00:02.324434 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:02.324394 2580 generic.go:356] "Generic (PLEG): container finished" podID="5ad0e5fc-930e-478a-abb7-f2e4b2377239" containerID="abf7e7186f26da622c8c928065c69441fdd9c7a65d97189277526e4741315fe5" exitCode=0 Aug 04 17:00:02.324878 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:02.324473 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764380-j2kbr" event={"ID":"5ad0e5fc-930e-478a-abb7-f2e4b2377239","Type":"ContainerDied","Data":"abf7e7186f26da622c8c928065c69441fdd9c7a65d97189277526e4741315fe5"} Aug 04 17:00:02.326160 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:02.326134 2580 generic.go:356] "Generic (PLEG): container finished" podID="0288e786-b008-4b87-9598-1c1ed3644f6b" containerID="4637198c72289680dc1d144959b572e035d444111549607083d076f67d085c97" exitCode=0 Aug 04 17:00:02.326274 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:02.326212 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a125dd-29764380-sgtb7" event={"ID":"0288e786-b008-4b87-9598-1c1ed3644f6b","Type":"ContainerDied","Data":"4637198c72289680dc1d144959b572e035d444111549607083d076f67d085c97"} Aug 04 17:00:02.327774 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:02.327757 2580 generic.go:356] "Generic (PLEG): container finished" podID="6b7d3634-aa65-497a-909c-b0f048531805" containerID="9c0d751bbee17d0ea14bef33744f1a39db404cb22795075dc53a970c7b67964a" exitCode=0 Aug 04 17:00:02.327852 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:02.327808 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3589b4-29764380-9ttlw" event={"ID":"6b7d3634-aa65-497a-909c-b0f048531805","Type":"ContainerDied","Data":"9c0d751bbee17d0ea14bef33744f1a39db404cb22795075dc53a970c7b67964a"} Aug 04 17:00:03.483812 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:03.483778 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a125dd-29764380-sgtb7" Aug 04 17:00:03.526834 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:03.526813 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3589b4-29764380-9ttlw" Aug 04 17:00:03.530391 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:03.530369 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764380-j2kbr" Aug 04 17:00:04.337474 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:04.337441 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764380-j2kbr" Aug 04 17:00:04.337474 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:04.337465 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764380-j2kbr" event={"ID":"5ad0e5fc-930e-478a-abb7-f2e4b2377239","Type":"ContainerDied","Data":"b11ba7c3ebce3e9284e9019b1d60b99369d155cac34c0275dd68643a4effe7c9"} Aug 04 17:00:04.337720 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:04.337510 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b11ba7c3ebce3e9284e9019b1d60b99369d155cac34c0275dd68643a4effe7c9" Aug 04 17:00:04.338749 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:04.338729 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a125dd-29764380-sgtb7" Aug 04 17:00:04.338749 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:04.338738 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a125dd-29764380-sgtb7" event={"ID":"0288e786-b008-4b87-9598-1c1ed3644f6b","Type":"ContainerDied","Data":"bfbe9bc7875606a6989523a01edd49a909ab574e8855ee7d21961c3479b50a23"} Aug 04 17:00:04.338968 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:04.338769 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfbe9bc7875606a6989523a01edd49a909ab574e8855ee7d21961c3479b50a23" Aug 04 17:00:04.340097 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:04.340069 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3589b4-29764380-9ttlw" event={"ID":"6b7d3634-aa65-497a-909c-b0f048531805","Type":"ContainerDied","Data":"415a92f0347c86680acd0897ee687324a48838584a5fc327ac0b0ddfae2ff5aa"} Aug 04 17:00:04.340236 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:04.340104 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3589b4-29764380-9ttlw" Aug 04 17:00:04.340236 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:04.340109 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="415a92f0347c86680acd0897ee687324a48838584a5fc327ac0b0ddfae2ff5aa" Aug 04 17:00:10.265474 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:10.265443 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb_2ae85abe-7737-48f6-abea-f9445ace7164/payload-processing/0.log" Aug 04 17:00:15.700592 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.700559 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs"] Aug 04 17:00:15.700973 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.700959 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ad0e5fc-930e-478a-abb7-f2e4b2377239" containerName="cleanup" Aug 04 17:00:15.701020 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.700974 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ad0e5fc-930e-478a-abb7-f2e4b2377239" containerName="cleanup" Aug 04 17:00:15.701020 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.700991 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b7d3634-aa65-497a-909c-b0f048531805" containerName="cleanup" Aug 04 17:00:15.701020 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.700996 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b7d3634-aa65-497a-909c-b0f048531805" containerName="cleanup" Aug 04 17:00:15.701020 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.701011 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0288e786-b008-4b87-9598-1c1ed3644f6b" containerName="cleanup" Aug 04 17:00:15.701135 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.701026 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0288e786-b008-4b87-9598-1c1ed3644f6b" containerName="cleanup" Aug 04 17:00:15.701135 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.701085 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ad0e5fc-930e-478a-abb7-f2e4b2377239" containerName="cleanup" Aug 04 17:00:15.701135 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.701092 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0288e786-b008-4b87-9598-1c1ed3644f6b" containerName="cleanup" Aug 04 17:00:15.701135 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.701099 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b7d3634-aa65-497a-909c-b0f048531805" containerName="cleanup" Aug 04 17:00:15.704316 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.704297 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" Aug 04 17:00:15.709072 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.709052 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-72d790\"" Aug 04 17:00:15.717583 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.717565 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs"] Aug 04 17:00:15.740048 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.740021 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96"] Aug 04 17:00:15.743466 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.743446 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" Aug 04 17:00:15.746409 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.746387 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-72d790-dockercfg-8tpcb\"" Aug 04 17:00:15.754628 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.754609 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96"] Aug 04 17:00:15.807997 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.807964 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpcxn\" (UniqueName: \"kubernetes.io/projected/846199d7-562f-4fd0-8ad7-535ad88f8980-kube-api-access-wpcxn\") pod \"payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs\" (UID: \"846199d7-562f-4fd0-8ad7-535ad88f8980\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" Aug 04 17:00:15.808135 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.808023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a72d140e-ecf4-4aed-9bab-708811fa26e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96\" (UID: \"a72d140e-ecf4-4aed-9bab-708811fa26e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" Aug 04 17:00:15.808135 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.808120 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846199d7-562f-4fd0-8ad7-535ad88f8980-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs\" (UID: \"846199d7-562f-4fd0-8ad7-535ad88f8980\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" Aug 04 17:00:15.808217 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.808144 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6np74\" (UniqueName: \"kubernetes.io/projected/a72d140e-ecf4-4aed-9bab-708811fa26e0-kube-api-access-6np74\") pod \"payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96\" (UID: \"a72d140e-ecf4-4aed-9bab-708811fa26e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" Aug 04 17:00:15.908907 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.908870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846199d7-562f-4fd0-8ad7-535ad88f8980-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs\" (UID: \"846199d7-562f-4fd0-8ad7-535ad88f8980\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" Aug 04 17:00:15.908907 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.908913 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6np74\" (UniqueName: \"kubernetes.io/projected/a72d140e-ecf4-4aed-9bab-708811fa26e0-kube-api-access-6np74\") pod \"payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96\" (UID: \"a72d140e-ecf4-4aed-9bab-708811fa26e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" Aug 04 17:00:15.909131 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.908939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpcxn\" (UniqueName: \"kubernetes.io/projected/846199d7-562f-4fd0-8ad7-535ad88f8980-kube-api-access-wpcxn\") pod \"payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs\" (UID: \"846199d7-562f-4fd0-8ad7-535ad88f8980\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" Aug 04 17:00:15.909131 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.908975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a72d140e-ecf4-4aed-9bab-708811fa26e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96\" (UID: \"a72d140e-ecf4-4aed-9bab-708811fa26e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" Aug 04 17:00:15.909642 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.909623 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a72d140e-ecf4-4aed-9bab-708811fa26e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96\" (UID: \"a72d140e-ecf4-4aed-9bab-708811fa26e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" Aug 04 17:00:15.909689 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.909655 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846199d7-562f-4fd0-8ad7-535ad88f8980-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs\" (UID: \"846199d7-562f-4fd0-8ad7-535ad88f8980\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" Aug 04 17:00:15.916966 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.916939 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpcxn\" (UniqueName: \"kubernetes.io/projected/846199d7-562f-4fd0-8ad7-535ad88f8980-kube-api-access-wpcxn\") pod \"payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs\" (UID: \"846199d7-562f-4fd0-8ad7-535ad88f8980\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" Aug 04 17:00:15.917068 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:15.917010 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6np74\" (UniqueName: \"kubernetes.io/projected/a72d140e-ecf4-4aed-9bab-708811fa26e0-kube-api-access-6np74\") pod \"payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96\" (UID: \"a72d140e-ecf4-4aed-9bab-708811fa26e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" Aug 04 17:00:16.015300 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:16.015198 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" Aug 04 17:00:16.054407 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:16.054377 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" Aug 04 17:00:16.160839 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:16.160812 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs"] Aug 04 17:00:16.163352 ip-10-0-132-42 kubenswrapper[2580]: W0804 17:00:16.163324 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846199d7_562f_4fd0_8ad7_535ad88f8980.slice/crio-564951f6344e8fd60ee33f5352f43664900570e307aa7e3e53541d3d33f2aa54 WatchSource:0}: Error finding container 564951f6344e8fd60ee33f5352f43664900570e307aa7e3e53541d3d33f2aa54: Status 404 returned error can't find the container with id 564951f6344e8fd60ee33f5352f43664900570e307aa7e3e53541d3d33f2aa54 Aug 04 17:00:16.202090 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:16.202060 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96"] Aug 04 17:00:16.204023 ip-10-0-132-42 kubenswrapper[2580]: W0804 17:00:16.203999 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72d140e_ecf4_4aed_9bab_708811fa26e0.slice/crio-e3aed9c7552ed986eb9ac47d1a47f2a0cfc8874edd58737ae5ac3d9d2345c3b7 WatchSource:0}: Error finding container e3aed9c7552ed986eb9ac47d1a47f2a0cfc8874edd58737ae5ac3d9d2345c3b7: Status 404 returned error can't find the container with id e3aed9c7552ed986eb9ac47d1a47f2a0cfc8874edd58737ae5ac3d9d2345c3b7 Aug 04 17:00:16.401127 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:16.401097 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" event={"ID":"846199d7-562f-4fd0-8ad7-535ad88f8980","Type":"ContainerStarted","Data":"61c355b91d0fd5efa4f429ae7559b7e4233282908a843022321b407b6c4a13a9"} Aug 04 17:00:16.401318 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:16.401165 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" Aug 04 17:00:16.401318 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:16.401193 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" event={"ID":"846199d7-562f-4fd0-8ad7-535ad88f8980","Type":"ContainerStarted","Data":"564951f6344e8fd60ee33f5352f43664900570e307aa7e3e53541d3d33f2aa54"} Aug 04 17:00:16.402581 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:16.402556 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" event={"ID":"a72d140e-ecf4-4aed-9bab-708811fa26e0","Type":"ContainerStarted","Data":"d382effa2ecd3e273cb4f5e5d72b1cd0e62938e413d4eaf9e5762990f29e2a94"} Aug 04 17:00:16.402692 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:16.402599 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" event={"ID":"a72d140e-ecf4-4aed-9bab-708811fa26e0","Type":"ContainerStarted","Data":"e3aed9c7552ed986eb9ac47d1a47f2a0cfc8874edd58737ae5ac3d9d2345c3b7"} Aug 04 17:00:16.402692 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:16.402676 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" Aug 04 17:00:16.418511 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:16.418473 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" podStartSLOduration=1.418460083 podStartE2EDuration="1.418460083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:00:16.416488933 +0000 UTC m=+3517.560821203" watchObservedRunningTime="2026-08-04 17:00:16.418460083 +0000 UTC m=+3517.562792328" Aug 04 17:00:16.433042 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:16.433001 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" podStartSLOduration=1.4329876289999999 podStartE2EDuration="1.432987629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:00:16.432611604 +0000 UTC m=+3517.576943871" watchObservedRunningTime="2026-08-04 17:00:16.432987629 +0000 UTC m=+3517.577319917" Aug 04 17:00:20.738495 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:20.738459 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 17:00:27.410298 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:27.410253 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" Aug 04 17:00:27.410670 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:27.410335 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" Aug 04 17:00:29.160987 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:29.160946 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk"] Aug 04 17:00:29.164654 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:29.164630 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk" Aug 04 17:00:29.167370 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:29.167349 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 17:00:29.167485 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:29.167388 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4pxfk\"" Aug 04 17:00:29.172141 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:29.171875 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk"] Aug 04 17:00:29.316164 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:29.316135 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da08b049-ba32-4daa-a1e9-d76027d90806-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk\" (UID: \"da08b049-ba32-4daa-a1e9-d76027d90806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk" Aug 04 17:00:29.416776 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:29.416672 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da08b049-ba32-4daa-a1e9-d76027d90806-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk\" (UID: \"da08b049-ba32-4daa-a1e9-d76027d90806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk" Aug 04 17:00:29.417272 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:29.417254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da08b049-ba32-4daa-a1e9-d76027d90806-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk\" (UID: \"da08b049-ba32-4daa-a1e9-d76027d90806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk" Aug 04 17:00:29.478679 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:29.478646 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk" Aug 04 17:00:29.823015 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:29.822902 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk"] Aug 04 17:00:29.825670 ip-10-0-132-42 kubenswrapper[2580]: W0804 17:00:29.825637 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda08b049_ba32_4daa_a1e9_d76027d90806.slice/crio-8f2c7066aead6dbbaa0a2b2b421fadec8f3431da3c397972dc84e626d625b208 WatchSource:0}: Error finding container 8f2c7066aead6dbbaa0a2b2b421fadec8f3431da3c397972dc84e626d625b208: Status 404 returned error can't find the container with id 8f2c7066aead6dbbaa0a2b2b421fadec8f3431da3c397972dc84e626d625b208 Aug 04 17:00:30.476698 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:30.476667 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk" event={"ID":"da08b049-ba32-4daa-a1e9-d76027d90806","Type":"ContainerStarted","Data":"bcbde19c8bba62dd8fb4124adeb503252161ee493c5b5a0be914580eb88ebf9b"} Aug 04 17:00:30.477017 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:30.476727 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk" event={"ID":"da08b049-ba32-4daa-a1e9-d76027d90806","Type":"ContainerStarted","Data":"8f2c7066aead6dbbaa0a2b2b421fadec8f3431da3c397972dc84e626d625b208"} Aug 04 17:00:31.483547 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:31.483509 2580 generic.go:356] "Generic (PLEG): container finished" podID="da08b049-ba32-4daa-a1e9-d76027d90806" containerID="bcbde19c8bba62dd8fb4124adeb503252161ee493c5b5a0be914580eb88ebf9b" exitCode=0 Aug 04 17:00:31.483920 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:31.483594 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk" event={"ID":"da08b049-ba32-4daa-a1e9-d76027d90806","Type":"ContainerDied","Data":"bcbde19c8bba62dd8fb4124adeb503252161ee493c5b5a0be914580eb88ebf9b"} Aug 04 17:00:31.615441 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:31.615415 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk" Aug 04 17:00:31.737192 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:31.737109 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da08b049-ba32-4daa-a1e9-d76027d90806-maas-api-service-ca\") pod \"da08b049-ba32-4daa-a1e9-d76027d90806\" (UID: \"da08b049-ba32-4daa-a1e9-d76027d90806\") " Aug 04 17:00:31.737434 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:31.737410 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da08b049-ba32-4daa-a1e9-d76027d90806-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da08b049-ba32-4daa-a1e9-d76027d90806" (UID: "da08b049-ba32-4daa-a1e9-d76027d90806"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:00:31.838182 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:31.838156 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da08b049-ba32-4daa-a1e9-d76027d90806-maas-api-service-ca\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:00:32.489631 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:32.489599 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk" Aug 04 17:00:32.490031 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:32.489640 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk" event={"ID":"da08b049-ba32-4daa-a1e9-d76027d90806","Type":"ContainerDied","Data":"8f2c7066aead6dbbaa0a2b2b421fadec8f3431da3c397972dc84e626d625b208"} Aug 04 17:00:32.490031 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:32.489700 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f2c7066aead6dbbaa0a2b2b421fadec8f3431da3c397972dc84e626d625b208" Aug 04 17:00:34.554465 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:34.554438 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96"] Aug 04 17:00:34.554891 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:34.554651 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" podUID="a72d140e-ecf4-4aed-9bab-708811fa26e0" containerName="payload-processing" containerID="cri-o://d382effa2ecd3e273cb4f5e5d72b1cd0e62938e413d4eaf9e5762990f29e2a94" gracePeriod=30 Aug 04 17:00:34.563991 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:34.563795 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs"] Aug 04 17:00:34.564150 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:34.564011 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" podUID="846199d7-562f-4fd0-8ad7-535ad88f8980" containerName="payload-pre-processing" containerID="cri-o://61c355b91d0fd5efa4f429ae7559b7e4233282908a843022321b407b6c4a13a9" gracePeriod=30 Aug 04 17:00:34.812070 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:34.811981 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk"] Aug 04 17:00:34.813932 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:34.813905 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-72d790-cl9sk"] Aug 04 17:00:34.847664 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:34.847642 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" Aug 04 17:00:34.850547 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:34.850528 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" Aug 04 17:00:34.965446 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:34.965419 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6np74\" (UniqueName: \"kubernetes.io/projected/a72d140e-ecf4-4aed-9bab-708811fa26e0-kube-api-access-6np74\") pod \"a72d140e-ecf4-4aed-9bab-708811fa26e0\" (UID: \"a72d140e-ecf4-4aed-9bab-708811fa26e0\") " Aug 04 17:00:34.965623 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:34.965481 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpcxn\" (UniqueName: \"kubernetes.io/projected/846199d7-562f-4fd0-8ad7-535ad88f8980-kube-api-access-wpcxn\") pod \"846199d7-562f-4fd0-8ad7-535ad88f8980\" (UID: \"846199d7-562f-4fd0-8ad7-535ad88f8980\") " Aug 04 17:00:34.965623 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:34.965501 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a72d140e-ecf4-4aed-9bab-708811fa26e0-plugins-config-volume\") pod \"a72d140e-ecf4-4aed-9bab-708811fa26e0\" (UID: \"a72d140e-ecf4-4aed-9bab-708811fa26e0\") " Aug 04 17:00:34.965623 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:34.965538 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846199d7-562f-4fd0-8ad7-535ad88f8980-plugins-config-volume\") pod \"846199d7-562f-4fd0-8ad7-535ad88f8980\" (UID: \"846199d7-562f-4fd0-8ad7-535ad88f8980\") " Aug 04 17:00:34.965880 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:34.965852 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72d140e-ecf4-4aed-9bab-708811fa26e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a72d140e-ecf4-4aed-9bab-708811fa26e0" (UID: "a72d140e-ecf4-4aed-9bab-708811fa26e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:00:34.966003 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:34.965920 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846199d7-562f-4fd0-8ad7-535ad88f8980-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "846199d7-562f-4fd0-8ad7-535ad88f8980" (UID: "846199d7-562f-4fd0-8ad7-535ad88f8980"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:00:34.967596 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:34.967570 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72d140e-ecf4-4aed-9bab-708811fa26e0-kube-api-access-6np74" (OuterVolumeSpecName: "kube-api-access-6np74") pod "a72d140e-ecf4-4aed-9bab-708811fa26e0" (UID: "a72d140e-ecf4-4aed-9bab-708811fa26e0"). InnerVolumeSpecName "kube-api-access-6np74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:00:34.967700 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:34.967642 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846199d7-562f-4fd0-8ad7-535ad88f8980-kube-api-access-wpcxn" (OuterVolumeSpecName: "kube-api-access-wpcxn") pod "846199d7-562f-4fd0-8ad7-535ad88f8980" (UID: "846199d7-562f-4fd0-8ad7-535ad88f8980"). InnerVolumeSpecName "kube-api-access-wpcxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:00:35.066849 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.066787 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6np74\" (UniqueName: \"kubernetes.io/projected/a72d140e-ecf4-4aed-9bab-708811fa26e0-kube-api-access-6np74\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:00:35.066849 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.066813 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpcxn\" (UniqueName: \"kubernetes.io/projected/846199d7-562f-4fd0-8ad7-535ad88f8980-kube-api-access-wpcxn\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:00:35.066849 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.066822 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a72d140e-ecf4-4aed-9bab-708811fa26e0-plugins-config-volume\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:00:35.066849 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.066831 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846199d7-562f-4fd0-8ad7-535ad88f8980-plugins-config-volume\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:00:35.319172 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.319097 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da08b049-ba32-4daa-a1e9-d76027d90806" path="/var/lib/kubelet/pods/da08b049-ba32-4daa-a1e9-d76027d90806/volumes" Aug 04 17:00:35.507715 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.507680 2580 generic.go:356] "Generic (PLEG): container finished" podID="846199d7-562f-4fd0-8ad7-535ad88f8980" containerID="61c355b91d0fd5efa4f429ae7559b7e4233282908a843022321b407b6c4a13a9" exitCode=0 Aug 04 17:00:35.507868 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.507722 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" Aug 04 17:00:35.507868 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.507756 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" event={"ID":"846199d7-562f-4fd0-8ad7-535ad88f8980","Type":"ContainerDied","Data":"61c355b91d0fd5efa4f429ae7559b7e4233282908a843022321b407b6c4a13a9"} Aug 04 17:00:35.507868 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.507804 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs" event={"ID":"846199d7-562f-4fd0-8ad7-535ad88f8980","Type":"ContainerDied","Data":"564951f6344e8fd60ee33f5352f43664900570e307aa7e3e53541d3d33f2aa54"} Aug 04 17:00:35.507868 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.507828 2580 scope.go:117] "RemoveContainer" containerID="61c355b91d0fd5efa4f429ae7559b7e4233282908a843022321b407b6c4a13a9" Aug 04 17:00:35.509196 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.509178 2580 generic.go:356] "Generic (PLEG): container finished" podID="a72d140e-ecf4-4aed-9bab-708811fa26e0" containerID="d382effa2ecd3e273cb4f5e5d72b1cd0e62938e413d4eaf9e5762990f29e2a94" exitCode=0 Aug 04 17:00:35.509345 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.509216 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" Aug 04 17:00:35.509345 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.509231 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" event={"ID":"a72d140e-ecf4-4aed-9bab-708811fa26e0","Type":"ContainerDied","Data":"d382effa2ecd3e273cb4f5e5d72b1cd0e62938e413d4eaf9e5762990f29e2a94"} Aug 04 17:00:35.509345 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.509266 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96" event={"ID":"a72d140e-ecf4-4aed-9bab-708811fa26e0","Type":"ContainerDied","Data":"e3aed9c7552ed986eb9ac47d1a47f2a0cfc8874edd58737ae5ac3d9d2345c3b7"} Aug 04 17:00:35.516902 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.516885 2580 scope.go:117] "RemoveContainer" containerID="61c355b91d0fd5efa4f429ae7559b7e4233282908a843022321b407b6c4a13a9" Aug 04 17:00:35.517161 ip-10-0-132-42 kubenswrapper[2580]: E0804 17:00:35.517143 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c355b91d0fd5efa4f429ae7559b7e4233282908a843022321b407b6c4a13a9\": container with ID starting with 61c355b91d0fd5efa4f429ae7559b7e4233282908a843022321b407b6c4a13a9 not found: ID does not exist" containerID="61c355b91d0fd5efa4f429ae7559b7e4233282908a843022321b407b6c4a13a9" Aug 04 17:00:35.517210 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.517170 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c355b91d0fd5efa4f429ae7559b7e4233282908a843022321b407b6c4a13a9"} err="failed to get container status \"61c355b91d0fd5efa4f429ae7559b7e4233282908a843022321b407b6c4a13a9\": rpc error: code = NotFound desc = could not find container \"61c355b91d0fd5efa4f429ae7559b7e4233282908a843022321b407b6c4a13a9\": container with ID starting with 61c355b91d0fd5efa4f429ae7559b7e4233282908a843022321b407b6c4a13a9 not found: ID does not exist" Aug 04 17:00:35.517210 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.517189 2580 scope.go:117] "RemoveContainer" containerID="d382effa2ecd3e273cb4f5e5d72b1cd0e62938e413d4eaf9e5762990f29e2a94" Aug 04 17:00:35.525135 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.525121 2580 scope.go:117] "RemoveContainer" containerID="d382effa2ecd3e273cb4f5e5d72b1cd0e62938e413d4eaf9e5762990f29e2a94" Aug 04 17:00:35.525562 ip-10-0-132-42 kubenswrapper[2580]: E0804 17:00:35.525537 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d382effa2ecd3e273cb4f5e5d72b1cd0e62938e413d4eaf9e5762990f29e2a94\": container with ID starting with d382effa2ecd3e273cb4f5e5d72b1cd0e62938e413d4eaf9e5762990f29e2a94 not found: ID does not exist" containerID="d382effa2ecd3e273cb4f5e5d72b1cd0e62938e413d4eaf9e5762990f29e2a94" Aug 04 17:00:35.525702 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.525679 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d382effa2ecd3e273cb4f5e5d72b1cd0e62938e413d4eaf9e5762990f29e2a94"} err="failed to get container status \"d382effa2ecd3e273cb4f5e5d72b1cd0e62938e413d4eaf9e5762990f29e2a94\": rpc error: code = NotFound desc = could not find container \"d382effa2ecd3e273cb4f5e5d72b1cd0e62938e413d4eaf9e5762990f29e2a94\": container with ID starting with d382effa2ecd3e273cb4f5e5d72b1cd0e62938e413d4eaf9e5762990f29e2a94 not found: ID does not exist" Aug 04 17:00:35.528114 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.528098 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs"] Aug 04 17:00:35.530762 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.530743 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-72d790-58fcbc9f76-9sgxs"] Aug 04 17:00:35.540760 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.540739 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96"] Aug 04 17:00:35.544430 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:35.544408 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-72d790-79bd54564b-jhr96"] Aug 04 17:00:37.320525 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:37.320487 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="846199d7-562f-4fd0-8ad7-535ad88f8980" path="/var/lib/kubelet/pods/846199d7-562f-4fd0-8ad7-535ad88f8980/volumes" Aug 04 17:00:37.321020 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:37.321000 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a72d140e-ecf4-4aed-9bab-708811fa26e0" path="/var/lib/kubelet/pods/a72d140e-ecf4-4aed-9bab-708811fa26e0/volumes" Aug 04 17:00:37.627582 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:37.627548 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 17:00:45.330086 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.330052 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8"] Aug 04 17:00:45.330480 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.330463 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a72d140e-ecf4-4aed-9bab-708811fa26e0" containerName="payload-processing" Aug 04 17:00:45.330480 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.330476 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a72d140e-ecf4-4aed-9bab-708811fa26e0" containerName="payload-processing" Aug 04 17:00:45.330564 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.330490 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da08b049-ba32-4daa-a1e9-d76027d90806" containerName="revoke-keys" Aug 04 17:00:45.330564 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.330496 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da08b049-ba32-4daa-a1e9-d76027d90806" containerName="revoke-keys" Aug 04 17:00:45.330564 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.330510 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="846199d7-562f-4fd0-8ad7-535ad88f8980" containerName="payload-pre-processing" Aug 04 17:00:45.330564 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.330523 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="846199d7-562f-4fd0-8ad7-535ad88f8980" containerName="payload-pre-processing" Aug 04 17:00:45.330677 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.330595 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da08b049-ba32-4daa-a1e9-d76027d90806" containerName="revoke-keys" Aug 04 17:00:45.330677 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.330605 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a72d140e-ecf4-4aed-9bab-708811fa26e0" containerName="payload-processing" Aug 04 17:00:45.330677 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.330614 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="846199d7-562f-4fd0-8ad7-535ad88f8980" containerName="payload-pre-processing" Aug 04 17:00:45.334927 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.334910 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8" Aug 04 17:00:45.337668 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.337596 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 17:00:45.338884 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.338858 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4pxfk\"" Aug 04 17:00:45.341312 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.341276 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8"] Aug 04 17:00:45.349617 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.349594 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50b2ed6-a979-40e3-9cff-acb070f181ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8\" (UID: \"b50b2ed6-a979-40e3-9cff-acb070f181ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8" Aug 04 17:00:45.450729 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.450704 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50b2ed6-a979-40e3-9cff-acb070f181ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8\" (UID: \"b50b2ed6-a979-40e3-9cff-acb070f181ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8" Aug 04 17:00:45.451243 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.451222 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50b2ed6-a979-40e3-9cff-acb070f181ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8\" (UID: \"b50b2ed6-a979-40e3-9cff-acb070f181ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8" Aug 04 17:00:45.646869 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.646794 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8" Aug 04 17:00:46.004026 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:45.998697 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8"] Aug 04 17:00:46.005944 ip-10-0-132-42 kubenswrapper[2580]: W0804 17:00:46.005919 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50b2ed6_a979_40e3_9cff_acb070f181ec.slice/crio-98250f98de29b188aa8066bad564a2995da51aa2e40f9fd67d24e4fbdc1c278c WatchSource:0}: Error finding container 98250f98de29b188aa8066bad564a2995da51aa2e40f9fd67d24e4fbdc1c278c: Status 404 returned error can't find the container with id 98250f98de29b188aa8066bad564a2995da51aa2e40f9fd67d24e4fbdc1c278c Aug 04 17:00:46.567137 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:46.567064 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8" event={"ID":"b50b2ed6-a979-40e3-9cff-acb070f181ec","Type":"ContainerStarted","Data":"903f6d685601b3e01e8f71676bf98fbae6e7c1701c8b1d5c07056f04be1212d7"} Aug 04 17:00:46.567137 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:46.567131 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8" event={"ID":"b50b2ed6-a979-40e3-9cff-acb070f181ec","Type":"ContainerStarted","Data":"98250f98de29b188aa8066bad564a2995da51aa2e40f9fd67d24e4fbdc1c278c"} Aug 04 17:00:46.582439 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:46.582396 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8" podStartSLOduration=1.5823807410000001 podStartE2EDuration="1.582380741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:00:46.581880431 +0000 UTC m=+3547.726212696" watchObservedRunningTime="2026-08-04 17:00:46.582380741 +0000 UTC m=+3547.726712987" Aug 04 17:00:47.573936 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:47.573897 2580 generic.go:356] "Generic (PLEG): container finished" podID="b50b2ed6-a979-40e3-9cff-acb070f181ec" containerID="903f6d685601b3e01e8f71676bf98fbae6e7c1701c8b1d5c07056f04be1212d7" exitCode=0 Aug 04 17:00:47.574348 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:47.573945 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8" event={"ID":"b50b2ed6-a979-40e3-9cff-acb070f181ec","Type":"ContainerDied","Data":"903f6d685601b3e01e8f71676bf98fbae6e7c1701c8b1d5c07056f04be1212d7"} Aug 04 17:00:48.715231 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:48.715208 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8" Aug 04 17:00:48.774695 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:48.774670 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50b2ed6-a979-40e3-9cff-acb070f181ec-maas-api-service-ca\") pod \"b50b2ed6-a979-40e3-9cff-acb070f181ec\" (UID: \"b50b2ed6-a979-40e3-9cff-acb070f181ec\") " Aug 04 17:00:48.774986 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:48.774966 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50b2ed6-a979-40e3-9cff-acb070f181ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b50b2ed6-a979-40e3-9cff-acb070f181ec" (UID: "b50b2ed6-a979-40e3-9cff-acb070f181ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:00:48.875676 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:48.875596 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50b2ed6-a979-40e3-9cff-acb070f181ec-maas-api-service-ca\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:00:49.584420 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:49.584343 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8" Aug 04 17:00:49.584420 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:49.584369 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8" event={"ID":"b50b2ed6-a979-40e3-9cff-acb070f181ec","Type":"ContainerDied","Data":"98250f98de29b188aa8066bad564a2995da51aa2e40f9fd67d24e4fbdc1c278c"} Aug 04 17:00:49.584420 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:49.584414 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98250f98de29b188aa8066bad564a2995da51aa2e40f9fd67d24e4fbdc1c278c" Aug 04 17:00:55.738567 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:55.738523 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 17:00:58.432584 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:58.432550 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a125dd-29764380-sgtb7"] Aug 04 17:00:58.434435 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:58.434412 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a125dd-29764380-sgtb7"] Aug 04 17:00:58.463322 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:58.463292 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p"] Aug 04 17:00:58.463669 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:58.463635 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" podUID="79e31aaf-afb9-4aff-8052-c7f834e1832c" containerName="payload-pre-processing" containerID="cri-o://c5cdde52a577e0c346024b8e61cd32daa156e8f31aed0871d462c84609fb65be" gracePeriod=30 Aug 04 17:00:58.613708 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:58.613679 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8"] Aug 04 17:00:58.615654 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:58.615628 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a125dd-f45w8"] Aug 04 17:00:58.636919 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:58.636644 2580 generic.go:356] "Generic (PLEG): container finished" podID="79e31aaf-afb9-4aff-8052-c7f834e1832c" containerID="c5cdde52a577e0c346024b8e61cd32daa156e8f31aed0871d462c84609fb65be" exitCode=0 Aug 04 17:00:58.636919 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:58.636720 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" event={"ID":"79e31aaf-afb9-4aff-8052-c7f834e1832c","Type":"ContainerDied","Data":"c5cdde52a577e0c346024b8e61cd32daa156e8f31aed0871d462c84609fb65be"} Aug 04 17:00:58.718970 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:58.718945 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" Aug 04 17:00:58.758585 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:58.758560 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgm9t\" (UniqueName: \"kubernetes.io/projected/79e31aaf-afb9-4aff-8052-c7f834e1832c-kube-api-access-xgm9t\") pod \"79e31aaf-afb9-4aff-8052-c7f834e1832c\" (UID: \"79e31aaf-afb9-4aff-8052-c7f834e1832c\") " Aug 04 17:00:58.758692 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:58.758605 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e31aaf-afb9-4aff-8052-c7f834e1832c-plugins-config-volume\") pod \"79e31aaf-afb9-4aff-8052-c7f834e1832c\" (UID: \"79e31aaf-afb9-4aff-8052-c7f834e1832c\") " Aug 04 17:00:58.759082 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:58.759046 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e31aaf-afb9-4aff-8052-c7f834e1832c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "79e31aaf-afb9-4aff-8052-c7f834e1832c" (UID: "79e31aaf-afb9-4aff-8052-c7f834e1832c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:00:58.760623 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:58.760594 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e31aaf-afb9-4aff-8052-c7f834e1832c-kube-api-access-xgm9t" (OuterVolumeSpecName: "kube-api-access-xgm9t") pod "79e31aaf-afb9-4aff-8052-c7f834e1832c" (UID: "79e31aaf-afb9-4aff-8052-c7f834e1832c"). InnerVolumeSpecName "kube-api-access-xgm9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:00:58.859836 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:58.859814 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgm9t\" (UniqueName: \"kubernetes.io/projected/79e31aaf-afb9-4aff-8052-c7f834e1832c-kube-api-access-xgm9t\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:00:58.859836 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:58.859833 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e31aaf-afb9-4aff-8052-c7f834e1832c-plugins-config-volume\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:00:59.319088 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:59.319054 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0288e786-b008-4b87-9598-1c1ed3644f6b" path="/var/lib/kubelet/pods/0288e786-b008-4b87-9598-1c1ed3644f6b/volumes" Aug 04 17:00:59.319431 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:59.319412 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b50b2ed6-a979-40e3-9cff-acb070f181ec" path="/var/lib/kubelet/pods/b50b2ed6-a979-40e3-9cff-acb070f181ec/volumes" Aug 04 17:00:59.643105 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:59.643073 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" event={"ID":"79e31aaf-afb9-4aff-8052-c7f834e1832c","Type":"ContainerDied","Data":"a205019acb80e4ed22ae5aa441218d4f816b1329cd3b946f2efcc08ca69efba7"} Aug 04 17:00:59.643551 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:59.643131 2580 scope.go:117] "RemoveContainer" containerID="c5cdde52a577e0c346024b8e61cd32daa156e8f31aed0871d462c84609fb65be" Aug 04 17:00:59.643551 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:59.643087 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p" Aug 04 17:00:59.659964 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:59.659940 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p"] Aug 04 17:00:59.663953 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:00:59.663933 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a125dd-68445b444-htj9p"] Aug 04 17:01:01.320339 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:01.320308 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79e31aaf-afb9-4aff-8052-c7f834e1832c" path="/var/lib/kubelet/pods/79e31aaf-afb9-4aff-8052-c7f834e1832c/volumes" Aug 04 17:01:02.827916 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:02.827881 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 17:01:39.503962 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:39.503936 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 17:01:39.511103 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:39.511082 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 17:01:52.931976 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:52.931940 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3589b4-29764380-9ttlw"] Aug 04 17:01:52.934557 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:52.934526 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3589b4-29764380-9ttlw"] Aug 04 17:01:52.962947 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:52.962917 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb"] Aug 04 17:01:52.963234 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:52.963168 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" podUID="2ae85abe-7737-48f6-abea-f9445ace7164" containerName="payload-processing" containerID="cri-o://77055dfdae03752bdb883d91f40d101af8a29dcc3080585523650838d4469974" gracePeriod=30 Aug 04 17:01:52.970141 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:52.970116 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs"] Aug 04 17:01:52.970818 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:52.970475 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" podUID="477d5ef1-a101-4dec-b30c-aeba3bb607ae" containerName="payload-pre-processing" containerID="cri-o://826ec4e23c9aa40c5407a564dba8ad1682a544898b2b75b2a792532a96b193c6" gracePeriod=30 Aug 04 17:01:53.226791 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.226762 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" Aug 04 17:01:53.249386 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.249361 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" Aug 04 17:01:53.289690 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.289655 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477d5ef1-a101-4dec-b30c-aeba3bb607ae-plugins-config-volume\") pod \"477d5ef1-a101-4dec-b30c-aeba3bb607ae\" (UID: \"477d5ef1-a101-4dec-b30c-aeba3bb607ae\") " Aug 04 17:01:53.289858 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.289715 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ae85abe-7737-48f6-abea-f9445ace7164-plugins-config-volume\") pod \"2ae85abe-7737-48f6-abea-f9445ace7164\" (UID: \"2ae85abe-7737-48f6-abea-f9445ace7164\") " Aug 04 17:01:53.289858 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.289731 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czdd2\" (UniqueName: \"kubernetes.io/projected/2ae85abe-7737-48f6-abea-f9445ace7164-kube-api-access-czdd2\") pod \"2ae85abe-7737-48f6-abea-f9445ace7164\" (UID: \"2ae85abe-7737-48f6-abea-f9445ace7164\") " Aug 04 17:01:53.289858 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.289794 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44td7\" (UniqueName: \"kubernetes.io/projected/477d5ef1-a101-4dec-b30c-aeba3bb607ae-kube-api-access-44td7\") pod \"477d5ef1-a101-4dec-b30c-aeba3bb607ae\" (UID: \"477d5ef1-a101-4dec-b30c-aeba3bb607ae\") " Aug 04 17:01:53.290103 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.290075 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477d5ef1-a101-4dec-b30c-aeba3bb607ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "477d5ef1-a101-4dec-b30c-aeba3bb607ae" (UID: "477d5ef1-a101-4dec-b30c-aeba3bb607ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:01:53.290181 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.290086 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae85abe-7737-48f6-abea-f9445ace7164-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ae85abe-7737-48f6-abea-f9445ace7164" (UID: "2ae85abe-7737-48f6-abea-f9445ace7164"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:01:53.291827 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.291804 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae85abe-7737-48f6-abea-f9445ace7164-kube-api-access-czdd2" (OuterVolumeSpecName: "kube-api-access-czdd2") pod "2ae85abe-7737-48f6-abea-f9445ace7164" (UID: "2ae85abe-7737-48f6-abea-f9445ace7164"). InnerVolumeSpecName "kube-api-access-czdd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:01:53.291920 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.291854 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477d5ef1-a101-4dec-b30c-aeba3bb607ae-kube-api-access-44td7" (OuterVolumeSpecName: "kube-api-access-44td7") pod "477d5ef1-a101-4dec-b30c-aeba3bb607ae" (UID: "477d5ef1-a101-4dec-b30c-aeba3bb607ae"). InnerVolumeSpecName "kube-api-access-44td7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:01:53.321967 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.321945 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b7d3634-aa65-497a-909c-b0f048531805" path="/var/lib/kubelet/pods/6b7d3634-aa65-497a-909c-b0f048531805/volumes" Aug 04 17:01:53.390565 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.390541 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44td7\" (UniqueName: \"kubernetes.io/projected/477d5ef1-a101-4dec-b30c-aeba3bb607ae-kube-api-access-44td7\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:01:53.390677 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.390570 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477d5ef1-a101-4dec-b30c-aeba3bb607ae-plugins-config-volume\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:01:53.390677 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.390584 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ae85abe-7737-48f6-abea-f9445ace7164-plugins-config-volume\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:01:53.390677 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.390597 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czdd2\" (UniqueName: \"kubernetes.io/projected/2ae85abe-7737-48f6-abea-f9445ace7164-kube-api-access-czdd2\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:01:53.918644 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.918612 2580 generic.go:356] "Generic (PLEG): container finished" podID="2ae85abe-7737-48f6-abea-f9445ace7164" containerID="77055dfdae03752bdb883d91f40d101af8a29dcc3080585523650838d4469974" exitCode=0 Aug 04 17:01:53.918820 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.918666 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" Aug 04 17:01:53.918820 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.918691 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" event={"ID":"2ae85abe-7737-48f6-abea-f9445ace7164","Type":"ContainerDied","Data":"77055dfdae03752bdb883d91f40d101af8a29dcc3080585523650838d4469974"} Aug 04 17:01:53.918820 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.918757 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb" event={"ID":"2ae85abe-7737-48f6-abea-f9445ace7164","Type":"ContainerDied","Data":"c3777a95df0703e5312b16156be7620883b97651c4d85fd7e4f9283d730b0d38"} Aug 04 17:01:53.918820 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.918786 2580 scope.go:117] "RemoveContainer" containerID="77055dfdae03752bdb883d91f40d101af8a29dcc3080585523650838d4469974" Aug 04 17:01:53.920080 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.920058 2580 generic.go:356] "Generic (PLEG): container finished" podID="477d5ef1-a101-4dec-b30c-aeba3bb607ae" containerID="826ec4e23c9aa40c5407a564dba8ad1682a544898b2b75b2a792532a96b193c6" exitCode=0 Aug 04 17:01:53.920182 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.920119 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" event={"ID":"477d5ef1-a101-4dec-b30c-aeba3bb607ae","Type":"ContainerDied","Data":"826ec4e23c9aa40c5407a564dba8ad1682a544898b2b75b2a792532a96b193c6"} Aug 04 17:01:53.920182 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.920155 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" event={"ID":"477d5ef1-a101-4dec-b30c-aeba3bb607ae","Type":"ContainerDied","Data":"79e53667c01bb20b020f464db47f6e1439550fb94bef1945b8158f903ab04300"} Aug 04 17:01:53.920182 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.920123 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs" Aug 04 17:01:53.927709 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.927693 2580 scope.go:117] "RemoveContainer" containerID="77055dfdae03752bdb883d91f40d101af8a29dcc3080585523650838d4469974" Aug 04 17:01:53.928018 ip-10-0-132-42 kubenswrapper[2580]: E0804 17:01:53.927998 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77055dfdae03752bdb883d91f40d101af8a29dcc3080585523650838d4469974\": container with ID starting with 77055dfdae03752bdb883d91f40d101af8a29dcc3080585523650838d4469974 not found: ID does not exist" containerID="77055dfdae03752bdb883d91f40d101af8a29dcc3080585523650838d4469974" Aug 04 17:01:53.928076 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.928027 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77055dfdae03752bdb883d91f40d101af8a29dcc3080585523650838d4469974"} err="failed to get container status \"77055dfdae03752bdb883d91f40d101af8a29dcc3080585523650838d4469974\": rpc error: code = NotFound desc = could not find container \"77055dfdae03752bdb883d91f40d101af8a29dcc3080585523650838d4469974\": container with ID starting with 77055dfdae03752bdb883d91f40d101af8a29dcc3080585523650838d4469974 not found: ID does not exist" Aug 04 17:01:53.928076 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.928060 2580 scope.go:117] "RemoveContainer" containerID="826ec4e23c9aa40c5407a564dba8ad1682a544898b2b75b2a792532a96b193c6" Aug 04 17:01:53.938011 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.937989 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs"] Aug 04 17:01:53.938364 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.938277 2580 scope.go:117] "RemoveContainer" containerID="826ec4e23c9aa40c5407a564dba8ad1682a544898b2b75b2a792532a96b193c6" Aug 04 17:01:53.938585 ip-10-0-132-42 kubenswrapper[2580]: E0804 17:01:53.938550 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826ec4e23c9aa40c5407a564dba8ad1682a544898b2b75b2a792532a96b193c6\": container with ID starting with 826ec4e23c9aa40c5407a564dba8ad1682a544898b2b75b2a792532a96b193c6 not found: ID does not exist" containerID="826ec4e23c9aa40c5407a564dba8ad1682a544898b2b75b2a792532a96b193c6" Aug 04 17:01:53.938643 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.938594 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826ec4e23c9aa40c5407a564dba8ad1682a544898b2b75b2a792532a96b193c6"} err="failed to get container status \"826ec4e23c9aa40c5407a564dba8ad1682a544898b2b75b2a792532a96b193c6\": rpc error: code = NotFound desc = could not find container \"826ec4e23c9aa40c5407a564dba8ad1682a544898b2b75b2a792532a96b193c6\": container with ID starting with 826ec4e23c9aa40c5407a564dba8ad1682a544898b2b75b2a792532a96b193c6 not found: ID does not exist" Aug 04 17:01:53.944556 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.944539 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3589b4-7df7d784dd-4wcxs"] Aug 04 17:01:53.954360 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.954339 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb"] Aug 04 17:01:53.957680 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:53.957663 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3589b4-54dfc9775d-ngxmb"] Aug 04 17:01:55.133623 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:55.133595 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 17:01:55.319258 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:55.319226 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ae85abe-7737-48f6-abea-f9445ace7164" path="/var/lib/kubelet/pods/2ae85abe-7737-48f6-abea-f9445ace7164/volumes" Aug 04 17:01:55.319571 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:01:55.319557 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="477d5ef1-a101-4dec-b30c-aeba3bb607ae" path="/var/lib/kubelet/pods/477d5ef1-a101-4dec-b30c-aeba3bb607ae/volumes" Aug 04 17:02:00.033577 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.033544 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc"] Aug 04 17:02:00.033950 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.033893 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae85abe-7737-48f6-abea-f9445ace7164" containerName="payload-processing" Aug 04 17:02:00.033950 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.033904 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae85abe-7737-48f6-abea-f9445ace7164" containerName="payload-processing" Aug 04 17:02:00.033950 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.033923 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b50b2ed6-a979-40e3-9cff-acb070f181ec" containerName="revoke-keys" Aug 04 17:02:00.033950 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.033929 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b50b2ed6-a979-40e3-9cff-acb070f181ec" containerName="revoke-keys" Aug 04 17:02:00.033950 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.033939 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e31aaf-afb9-4aff-8052-c7f834e1832c" containerName="payload-pre-processing" Aug 04 17:02:00.033950 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.033945 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e31aaf-afb9-4aff-8052-c7f834e1832c" containerName="payload-pre-processing" Aug 04 17:02:00.034145 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.033959 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="477d5ef1-a101-4dec-b30c-aeba3bb607ae" containerName="payload-pre-processing" Aug 04 17:02:00.034145 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.033964 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="477d5ef1-a101-4dec-b30c-aeba3bb607ae" containerName="payload-pre-processing" Aug 04 17:02:00.034145 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.034016 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e31aaf-afb9-4aff-8052-c7f834e1832c" containerName="payload-pre-processing" Aug 04 17:02:00.034145 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.034023 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ae85abe-7737-48f6-abea-f9445ace7164" containerName="payload-processing" Aug 04 17:02:00.034145 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.034029 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="477d5ef1-a101-4dec-b30c-aeba3bb607ae" containerName="payload-pre-processing" Aug 04 17:02:00.034145 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.034035 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b50b2ed6-a979-40e3-9cff-acb070f181ec" containerName="revoke-keys" Aug 04 17:02:00.038256 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.038238 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc" Aug 04 17:02:00.041021 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.040997 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4pxfk\"" Aug 04 17:02:00.041021 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.041017 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 17:02:00.044600 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.044578 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc"] Aug 04 17:02:00.138314 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.138276 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f234c6-a449-4208-a08f-a10b38933a5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc\" (UID: \"c1f234c6-a449-4208-a08f-a10b38933a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc" Aug 04 17:02:00.238885 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.238860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f234c6-a449-4208-a08f-a10b38933a5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc\" (UID: \"c1f234c6-a449-4208-a08f-a10b38933a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc" Aug 04 17:02:00.239424 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.239406 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f234c6-a449-4208-a08f-a10b38933a5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc\" (UID: \"c1f234c6-a449-4208-a08f-a10b38933a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc" Aug 04 17:02:00.350420 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.350398 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc" Aug 04 17:02:00.493703 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.493639 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc"] Aug 04 17:02:00.496302 ip-10-0-132-42 kubenswrapper[2580]: W0804 17:02:00.496253 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f234c6_a449_4208_a08f_a10b38933a5a.slice/crio-e87848f43db3597457e60ed143f4b534fe8610b06e94f0360aea3b1dae0f88ee WatchSource:0}: Error finding container e87848f43db3597457e60ed143f4b534fe8610b06e94f0360aea3b1dae0f88ee: Status 404 returned error can't find the container with id e87848f43db3597457e60ed143f4b534fe8610b06e94f0360aea3b1dae0f88ee Aug 04 17:02:00.959201 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.959113 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc" event={"ID":"c1f234c6-a449-4208-a08f-a10b38933a5a","Type":"ContainerStarted","Data":"6624df392f0f74d50865ea645395967a791fe571f429783a14b1df01206561c7"} Aug 04 17:02:00.959201 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.959174 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc" event={"ID":"c1f234c6-a449-4208-a08f-a10b38933a5a","Type":"ContainerStarted","Data":"e87848f43db3597457e60ed143f4b534fe8610b06e94f0360aea3b1dae0f88ee"} Aug 04 17:02:00.979872 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:00.979816 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc" podStartSLOduration=0.979795149 podStartE2EDuration="979.795149ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:02:00.979294082 +0000 UTC m=+3622.123626340" watchObservedRunningTime="2026-08-04 17:02:00.979795149 +0000 UTC m=+3622.124127399" Aug 04 17:02:01.965600 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:01.965567 2580 generic.go:356] "Generic (PLEG): container finished" podID="c1f234c6-a449-4208-a08f-a10b38933a5a" containerID="6624df392f0f74d50865ea645395967a791fe571f429783a14b1df01206561c7" exitCode=0 Aug 04 17:02:01.965980 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:01.965656 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc" event={"ID":"c1f234c6-a449-4208-a08f-a10b38933a5a","Type":"ContainerDied","Data":"6624df392f0f74d50865ea645395967a791fe571f429783a14b1df01206561c7"} Aug 04 17:02:03.122896 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:03.122867 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc" Aug 04 17:02:03.164450 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:03.164428 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f234c6-a449-4208-a08f-a10b38933a5a-maas-api-service-ca\") pod \"c1f234c6-a449-4208-a08f-a10b38933a5a\" (UID: \"c1f234c6-a449-4208-a08f-a10b38933a5a\") " Aug 04 17:02:03.164743 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:03.164722 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f234c6-a449-4208-a08f-a10b38933a5a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1f234c6-a449-4208-a08f-a10b38933a5a" (UID: "c1f234c6-a449-4208-a08f-a10b38933a5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:02:03.265930 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:03.265870 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f234c6-a449-4208-a08f-a10b38933a5a-maas-api-service-ca\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:02:03.976749 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:03.976718 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc" Aug 04 17:02:03.976903 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:03.976739 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc" event={"ID":"c1f234c6-a449-4208-a08f-a10b38933a5a","Type":"ContainerDied","Data":"e87848f43db3597457e60ed143f4b534fe8610b06e94f0360aea3b1dae0f88ee"} Aug 04 17:02:03.976903 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:03.976822 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e87848f43db3597457e60ed143f4b534fe8610b06e94f0360aea3b1dae0f88ee" Aug 04 17:02:10.625757 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:10.625720 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh"] Aug 04 17:02:10.626267 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:10.625954 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" podUID="2371e1d3-86fc-43cd-879d-0baa283e5eac" containerName="payload-processing" containerID="cri-o://07e0fcbc5fb2742dc7365fe15ef57f05dcc9eefb1fffed6adc52cc10a0c145eb" gracePeriod=30 Aug 04 17:02:10.634568 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:10.634334 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h"] Aug 04 17:02:10.634647 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:10.634568 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" podUID="f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb" containerName="payload-pre-processing" containerID="cri-o://3a4d7d24858da5364de5aac884c5fcba1261a5f45b91bb2086d3db4a5eed60c0" gracePeriod=30 Aug 04 17:02:10.760029 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:10.759999 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc"] Aug 04 17:02:10.761843 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:10.761816 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4e1c51-k56kc"] Aug 04 17:02:10.907912 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:10.907885 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" Aug 04 17:02:10.910877 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:10.910853 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" Aug 04 17:02:11.012204 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.012171 2580 generic.go:356] "Generic (PLEG): container finished" podID="f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb" containerID="3a4d7d24858da5364de5aac884c5fcba1261a5f45b91bb2086d3db4a5eed60c0" exitCode=0 Aug 04 17:02:11.012385 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.012215 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" Aug 04 17:02:11.012385 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.012262 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" event={"ID":"f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb","Type":"ContainerDied","Data":"3a4d7d24858da5364de5aac884c5fcba1261a5f45b91bb2086d3db4a5eed60c0"} Aug 04 17:02:11.012385 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.012336 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h" event={"ID":"f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb","Type":"ContainerDied","Data":"06af82a18b42e465a455c068c9b01aa91faedfa90db3a3dd2fb7cf718f7ce68d"} Aug 04 17:02:11.012385 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.012362 2580 scope.go:117] "RemoveContainer" containerID="3a4d7d24858da5364de5aac884c5fcba1261a5f45b91bb2086d3db4a5eed60c0" Aug 04 17:02:11.013670 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.013650 2580 generic.go:356] "Generic (PLEG): container finished" podID="2371e1d3-86fc-43cd-879d-0baa283e5eac" containerID="07e0fcbc5fb2742dc7365fe15ef57f05dcc9eefb1fffed6adc52cc10a0c145eb" exitCode=0 Aug 04 17:02:11.013768 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.013696 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" Aug 04 17:02:11.013835 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.013698 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" event={"ID":"2371e1d3-86fc-43cd-879d-0baa283e5eac","Type":"ContainerDied","Data":"07e0fcbc5fb2742dc7365fe15ef57f05dcc9eefb1fffed6adc52cc10a0c145eb"} Aug 04 17:02:11.013835 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.013828 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh" event={"ID":"2371e1d3-86fc-43cd-879d-0baa283e5eac","Type":"ContainerDied","Data":"b5017ccff49319d0621f54e21a52c85eaad038a7c600a784f1fb55850bf2a8c0"} Aug 04 17:02:11.020858 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.020842 2580 scope.go:117] "RemoveContainer" containerID="3a4d7d24858da5364de5aac884c5fcba1261a5f45b91bb2086d3db4a5eed60c0" Aug 04 17:02:11.021095 ip-10-0-132-42 kubenswrapper[2580]: E0804 17:02:11.021078 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a4d7d24858da5364de5aac884c5fcba1261a5f45b91bb2086d3db4a5eed60c0\": container with ID starting with 3a4d7d24858da5364de5aac884c5fcba1261a5f45b91bb2086d3db4a5eed60c0 not found: ID does not exist" containerID="3a4d7d24858da5364de5aac884c5fcba1261a5f45b91bb2086d3db4a5eed60c0" Aug 04 17:02:11.021134 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.021102 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a4d7d24858da5364de5aac884c5fcba1261a5f45b91bb2086d3db4a5eed60c0"} err="failed to get container status \"3a4d7d24858da5364de5aac884c5fcba1261a5f45b91bb2086d3db4a5eed60c0\": rpc error: code = NotFound desc = could not find container \"3a4d7d24858da5364de5aac884c5fcba1261a5f45b91bb2086d3db4a5eed60c0\": container with ID starting with 3a4d7d24858da5364de5aac884c5fcba1261a5f45b91bb2086d3db4a5eed60c0 not found: ID does not exist" Aug 04 17:02:11.021134 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.021118 2580 scope.go:117] "RemoveContainer" containerID="07e0fcbc5fb2742dc7365fe15ef57f05dcc9eefb1fffed6adc52cc10a0c145eb" Aug 04 17:02:11.021251 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.021234 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msmpw\" (UniqueName: \"kubernetes.io/projected/f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb-kube-api-access-msmpw\") pod \"f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb\" (UID: \"f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb\") " Aug 04 17:02:11.021333 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.021274 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2371e1d3-86fc-43cd-879d-0baa283e5eac-plugins-config-volume\") pod \"2371e1d3-86fc-43cd-879d-0baa283e5eac\" (UID: \"2371e1d3-86fc-43cd-879d-0baa283e5eac\") " Aug 04 17:02:11.021333 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.021312 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb-plugins-config-volume\") pod \"f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb\" (UID: \"f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb\") " Aug 04 17:02:11.021440 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.021357 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrpx2\" (UniqueName: \"kubernetes.io/projected/2371e1d3-86fc-43cd-879d-0baa283e5eac-kube-api-access-vrpx2\") pod \"2371e1d3-86fc-43cd-879d-0baa283e5eac\" (UID: \"2371e1d3-86fc-43cd-879d-0baa283e5eac\") " Aug 04 17:02:11.021646 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.021628 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2371e1d3-86fc-43cd-879d-0baa283e5eac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2371e1d3-86fc-43cd-879d-0baa283e5eac" (UID: "2371e1d3-86fc-43cd-879d-0baa283e5eac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:02:11.021646 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.021636 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb" (UID: "f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:02:11.023300 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.023252 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb-kube-api-access-msmpw" (OuterVolumeSpecName: "kube-api-access-msmpw") pod "f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb" (UID: "f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb"). InnerVolumeSpecName "kube-api-access-msmpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:02:11.023526 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.023502 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2371e1d3-86fc-43cd-879d-0baa283e5eac-kube-api-access-vrpx2" (OuterVolumeSpecName: "kube-api-access-vrpx2") pod "2371e1d3-86fc-43cd-879d-0baa283e5eac" (UID: "2371e1d3-86fc-43cd-879d-0baa283e5eac"). InnerVolumeSpecName "kube-api-access-vrpx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:02:11.034421 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.034404 2580 scope.go:117] "RemoveContainer" containerID="07e0fcbc5fb2742dc7365fe15ef57f05dcc9eefb1fffed6adc52cc10a0c145eb" Aug 04 17:02:11.034670 ip-10-0-132-42 kubenswrapper[2580]: E0804 17:02:11.034648 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e0fcbc5fb2742dc7365fe15ef57f05dcc9eefb1fffed6adc52cc10a0c145eb\": container with ID starting with 07e0fcbc5fb2742dc7365fe15ef57f05dcc9eefb1fffed6adc52cc10a0c145eb not found: ID does not exist" containerID="07e0fcbc5fb2742dc7365fe15ef57f05dcc9eefb1fffed6adc52cc10a0c145eb" Aug 04 17:02:11.034724 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.034672 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e0fcbc5fb2742dc7365fe15ef57f05dcc9eefb1fffed6adc52cc10a0c145eb"} err="failed to get container status \"07e0fcbc5fb2742dc7365fe15ef57f05dcc9eefb1fffed6adc52cc10a0c145eb\": rpc error: code = NotFound desc = could not find container \"07e0fcbc5fb2742dc7365fe15ef57f05dcc9eefb1fffed6adc52cc10a0c145eb\": container with ID starting with 07e0fcbc5fb2742dc7365fe15ef57f05dcc9eefb1fffed6adc52cc10a0c145eb not found: ID does not exist" Aug 04 17:02:11.121883 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.121864 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrpx2\" (UniqueName: \"kubernetes.io/projected/2371e1d3-86fc-43cd-879d-0baa283e5eac-kube-api-access-vrpx2\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:02:11.121883 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.121884 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msmpw\" (UniqueName: \"kubernetes.io/projected/f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb-kube-api-access-msmpw\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:02:11.122027 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.121895 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2371e1d3-86fc-43cd-879d-0baa283e5eac-plugins-config-volume\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:02:11.122027 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.121904 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb-plugins-config-volume\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:02:11.320870 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.320812 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1f234c6-a449-4208-a08f-a10b38933a5a" path="/var/lib/kubelet/pods/c1f234c6-a449-4208-a08f-a10b38933a5a/volumes" Aug 04 17:02:11.339922 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.339902 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h"] Aug 04 17:02:11.341588 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.341569 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4e1c51-7c4f7cc5b8-9b82h"] Aug 04 17:02:11.350669 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.350649 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh"] Aug 04 17:02:11.354672 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:11.354654 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4e1c51-7c7cc5d758-msxdh"] Aug 04 17:02:12.830801 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:12.830766 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 17:02:13.319902 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:13.319822 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2371e1d3-86fc-43cd-879d-0baa283e5eac" path="/var/lib/kubelet/pods/2371e1d3-86fc-43cd-879d-0baa283e5eac/volumes" Aug 04 17:02:13.320235 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:13.320217 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb" path="/var/lib/kubelet/pods/f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb/volumes" Aug 04 17:02:17.747438 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.747398 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-222eca-jzltb"] Aug 04 17:02:17.747924 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.747905 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1f234c6-a449-4208-a08f-a10b38933a5a" containerName="revoke-keys" Aug 04 17:02:17.748003 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.747927 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1f234c6-a449-4208-a08f-a10b38933a5a" containerName="revoke-keys" Aug 04 17:02:17.748003 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.747943 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2371e1d3-86fc-43cd-879d-0baa283e5eac" containerName="payload-processing" Aug 04 17:02:17.748003 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.747951 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2371e1d3-86fc-43cd-879d-0baa283e5eac" containerName="payload-processing" Aug 04 17:02:17.748003 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.747966 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb" containerName="payload-pre-processing" Aug 04 17:02:17.748003 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.747975 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb" containerName="payload-pre-processing" Aug 04 17:02:17.748241 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.748066 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2371e1d3-86fc-43cd-879d-0baa283e5eac" containerName="payload-processing" Aug 04 17:02:17.748241 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.748083 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1f234c6-a449-4208-a08f-a10b38933a5a" containerName="revoke-keys" Aug 04 17:02:17.748241 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.748092 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6a37ce6-8a0f-47b8-856e-e9cc9112f9bb" containerName="payload-pre-processing" Aug 04 17:02:17.752666 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.752644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-222eca-jzltb" Aug 04 17:02:17.755493 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.755475 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 17:02:17.755603 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.755514 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4pxfk\"" Aug 04 17:02:17.759371 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.759349 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-222eca-jzltb"] Aug 04 17:02:17.874205 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.874174 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052f9387-e170-40be-9a98-37e8cd2f0d09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-222eca-jzltb\" (UID: \"052f9387-e170-40be-9a98-37e8cd2f0d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-222eca-jzltb" Aug 04 17:02:17.975622 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.975590 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052f9387-e170-40be-9a98-37e8cd2f0d09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-222eca-jzltb\" (UID: \"052f9387-e170-40be-9a98-37e8cd2f0d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-222eca-jzltb" Aug 04 17:02:17.976129 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:17.976110 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052f9387-e170-40be-9a98-37e8cd2f0d09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-222eca-jzltb\" (UID: \"052f9387-e170-40be-9a98-37e8cd2f0d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-222eca-jzltb" Aug 04 17:02:18.064036 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:18.063966 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-222eca-jzltb" Aug 04 17:02:18.192528 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:18.192505 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-222eca-jzltb"] Aug 04 17:02:18.194396 ip-10-0-132-42 kubenswrapper[2580]: W0804 17:02:18.194364 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052f9387_e170_40be_9a98_37e8cd2f0d09.slice/crio-4434e8cca3983c8deb77d484cf8bcd2c9a106b33d3cb3b9ef86274e5ff2be7e5 WatchSource:0}: Error finding container 4434e8cca3983c8deb77d484cf8bcd2c9a106b33d3cb3b9ef86274e5ff2be7e5: Status 404 returned error can't find the container with id 4434e8cca3983c8deb77d484cf8bcd2c9a106b33d3cb3b9ef86274e5ff2be7e5 Aug 04 17:02:19.057333 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:19.057232 2580 generic.go:356] "Generic (PLEG): container finished" podID="052f9387-e170-40be-9a98-37e8cd2f0d09" containerID="04171a6693e47a08e5ef8c557a6d7cd4a34f22758154b3527bed7f7d1196e88f" exitCode=0 Aug 04 17:02:19.057672 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:19.057326 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-222eca-jzltb" event={"ID":"052f9387-e170-40be-9a98-37e8cd2f0d09","Type":"ContainerDied","Data":"04171a6693e47a08e5ef8c557a6d7cd4a34f22758154b3527bed7f7d1196e88f"} Aug 04 17:02:19.057672 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:19.057379 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-222eca-jzltb" event={"ID":"052f9387-e170-40be-9a98-37e8cd2f0d09","Type":"ContainerStarted","Data":"4434e8cca3983c8deb77d484cf8bcd2c9a106b33d3cb3b9ef86274e5ff2be7e5"} Aug 04 17:02:20.186807 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:20.186780 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-222eca-jzltb" Aug 04 17:02:20.294313 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:20.294276 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052f9387-e170-40be-9a98-37e8cd2f0d09-maas-api-service-ca\") pod \"052f9387-e170-40be-9a98-37e8cd2f0d09\" (UID: \"052f9387-e170-40be-9a98-37e8cd2f0d09\") " Aug 04 17:02:20.294616 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:20.294598 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052f9387-e170-40be-9a98-37e8cd2f0d09-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "052f9387-e170-40be-9a98-37e8cd2f0d09" (UID: "052f9387-e170-40be-9a98-37e8cd2f0d09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:02:20.395783 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:20.395756 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052f9387-e170-40be-9a98-37e8cd2f0d09-maas-api-service-ca\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:02:21.067951 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:21.067914 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-222eca-jzltb" event={"ID":"052f9387-e170-40be-9a98-37e8cd2f0d09","Type":"ContainerDied","Data":"4434e8cca3983c8deb77d484cf8bcd2c9a106b33d3cb3b9ef86274e5ff2be7e5"} Aug 04 17:02:21.068105 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:21.067962 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4434e8cca3983c8deb77d484cf8bcd2c9a106b33d3cb3b9ef86274e5ff2be7e5" Aug 04 17:02:21.068105 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:21.067923 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-222eca-jzltb" Aug 04 17:02:28.233549 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.233469 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz"] Aug 04 17:02:28.234006 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.233747 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" podUID="4f753cb0-606f-4bf6-a064-c5071fc35c61" containerName="payload-processing" containerID="cri-o://baaf5f94baa9f2b70f5160cb5bc80ec5b9cbbcb5c10211ceb07f8a728ec46152" gracePeriod=30 Aug 04 17:02:28.240338 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.240311 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95"] Aug 04 17:02:28.240826 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.240762 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" podUID="15434214-f271-4492-8d0e-71621239d77a" containerName="payload-pre-processing" containerID="cri-o://43eb054afbc5d61d87dffff5cd2d5cee31f65c6084808d24bd275e6e2b32a044" gracePeriod=30 Aug 04 17:02:28.373619 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.373591 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-222eca-jzltb"] Aug 04 17:02:28.379307 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.378220 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-222eca-jzltb"] Aug 04 17:02:28.503237 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.503217 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" Aug 04 17:02:28.527599 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.527572 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" Aug 04 17:02:28.558772 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.558752 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w28wr\" (UniqueName: \"kubernetes.io/projected/4f753cb0-606f-4bf6-a064-c5071fc35c61-kube-api-access-w28wr\") pod \"4f753cb0-606f-4bf6-a064-c5071fc35c61\" (UID: \"4f753cb0-606f-4bf6-a064-c5071fc35c61\") " Aug 04 17:02:28.558890 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.558798 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f753cb0-606f-4bf6-a064-c5071fc35c61-plugins-config-volume\") pod \"4f753cb0-606f-4bf6-a064-c5071fc35c61\" (UID: \"4f753cb0-606f-4bf6-a064-c5071fc35c61\") " Aug 04 17:02:28.559212 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.559192 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f753cb0-606f-4bf6-a064-c5071fc35c61-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f753cb0-606f-4bf6-a064-c5071fc35c61" (UID: "4f753cb0-606f-4bf6-a064-c5071fc35c61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:02:28.560667 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.560650 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f753cb0-606f-4bf6-a064-c5071fc35c61-kube-api-access-w28wr" (OuterVolumeSpecName: "kube-api-access-w28wr") pod "4f753cb0-606f-4bf6-a064-c5071fc35c61" (UID: "4f753cb0-606f-4bf6-a064-c5071fc35c61"). InnerVolumeSpecName "kube-api-access-w28wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:02:28.659605 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.659586 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dml2\" (UniqueName: \"kubernetes.io/projected/15434214-f271-4492-8d0e-71621239d77a-kube-api-access-8dml2\") pod \"15434214-f271-4492-8d0e-71621239d77a\" (UID: \"15434214-f271-4492-8d0e-71621239d77a\") " Aug 04 17:02:28.659713 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.659674 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15434214-f271-4492-8d0e-71621239d77a-plugins-config-volume\") pod \"15434214-f271-4492-8d0e-71621239d77a\" (UID: \"15434214-f271-4492-8d0e-71621239d77a\") " Aug 04 17:02:28.659865 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.659848 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w28wr\" (UniqueName: \"kubernetes.io/projected/4f753cb0-606f-4bf6-a064-c5071fc35c61-kube-api-access-w28wr\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:02:28.659902 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.659871 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f753cb0-606f-4bf6-a064-c5071fc35c61-plugins-config-volume\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:02:28.660010 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.659994 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15434214-f271-4492-8d0e-71621239d77a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15434214-f271-4492-8d0e-71621239d77a" (UID: "15434214-f271-4492-8d0e-71621239d77a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:02:28.661373 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.661348 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15434214-f271-4492-8d0e-71621239d77a-kube-api-access-8dml2" (OuterVolumeSpecName: "kube-api-access-8dml2") pod "15434214-f271-4492-8d0e-71621239d77a" (UID: "15434214-f271-4492-8d0e-71621239d77a"). InnerVolumeSpecName "kube-api-access-8dml2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:02:28.760437 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.760385 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dml2\" (UniqueName: \"kubernetes.io/projected/15434214-f271-4492-8d0e-71621239d77a-kube-api-access-8dml2\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:02:28.760437 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:28.760413 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15434214-f271-4492-8d0e-71621239d77a-plugins-config-volume\") on node \"ip-10-0-132-42.ec2.internal\" DevicePath \"\"" Aug 04 17:02:29.109452 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.109415 2580 generic.go:356] "Generic (PLEG): container finished" podID="4f753cb0-606f-4bf6-a064-c5071fc35c61" containerID="baaf5f94baa9f2b70f5160cb5bc80ec5b9cbbcb5c10211ceb07f8a728ec46152" exitCode=0 Aug 04 17:02:29.109617 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.109454 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" Aug 04 17:02:29.109617 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.109506 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" event={"ID":"4f753cb0-606f-4bf6-a064-c5071fc35c61","Type":"ContainerDied","Data":"baaf5f94baa9f2b70f5160cb5bc80ec5b9cbbcb5c10211ceb07f8a728ec46152"} Aug 04 17:02:29.109617 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.109557 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz" event={"ID":"4f753cb0-606f-4bf6-a064-c5071fc35c61","Type":"ContainerDied","Data":"3a05cbf7dd7e285a0227652b5105f73285e68e2885e6f99c29560913a4c200ee"} Aug 04 17:02:29.109617 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.109584 2580 scope.go:117] "RemoveContainer" containerID="baaf5f94baa9f2b70f5160cb5bc80ec5b9cbbcb5c10211ceb07f8a728ec46152" Aug 04 17:02:29.110921 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.110896 2580 generic.go:356] "Generic (PLEG): container finished" podID="15434214-f271-4492-8d0e-71621239d77a" containerID="43eb054afbc5d61d87dffff5cd2d5cee31f65c6084808d24bd275e6e2b32a044" exitCode=0 Aug 04 17:02:29.111032 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.110923 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" event={"ID":"15434214-f271-4492-8d0e-71621239d77a","Type":"ContainerDied","Data":"43eb054afbc5d61d87dffff5cd2d5cee31f65c6084808d24bd275e6e2b32a044"} Aug 04 17:02:29.111032 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.110940 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" Aug 04 17:02:29.111032 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.110991 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95" event={"ID":"15434214-f271-4492-8d0e-71621239d77a","Type":"ContainerDied","Data":"83548b6b7fb3793a7840e175360384badee4957f9cfa21d1bbec222fda734f18"} Aug 04 17:02:29.118802 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.118783 2580 scope.go:117] "RemoveContainer" containerID="baaf5f94baa9f2b70f5160cb5bc80ec5b9cbbcb5c10211ceb07f8a728ec46152" Aug 04 17:02:29.119088 ip-10-0-132-42 kubenswrapper[2580]: E0804 17:02:29.119066 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baaf5f94baa9f2b70f5160cb5bc80ec5b9cbbcb5c10211ceb07f8a728ec46152\": container with ID starting with baaf5f94baa9f2b70f5160cb5bc80ec5b9cbbcb5c10211ceb07f8a728ec46152 not found: ID does not exist" containerID="baaf5f94baa9f2b70f5160cb5bc80ec5b9cbbcb5c10211ceb07f8a728ec46152" Aug 04 17:02:29.119151 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.119096 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baaf5f94baa9f2b70f5160cb5bc80ec5b9cbbcb5c10211ceb07f8a728ec46152"} err="failed to get container status \"baaf5f94baa9f2b70f5160cb5bc80ec5b9cbbcb5c10211ceb07f8a728ec46152\": rpc error: code = NotFound desc = could not find container \"baaf5f94baa9f2b70f5160cb5bc80ec5b9cbbcb5c10211ceb07f8a728ec46152\": container with ID starting with baaf5f94baa9f2b70f5160cb5bc80ec5b9cbbcb5c10211ceb07f8a728ec46152 not found: ID does not exist" Aug 04 17:02:29.119151 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.119112 2580 scope.go:117] "RemoveContainer" containerID="43eb054afbc5d61d87dffff5cd2d5cee31f65c6084808d24bd275e6e2b32a044" Aug 04 17:02:29.127080 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.127061 2580 scope.go:117] "RemoveContainer" containerID="43eb054afbc5d61d87dffff5cd2d5cee31f65c6084808d24bd275e6e2b32a044" Aug 04 17:02:29.127316 ip-10-0-132-42 kubenswrapper[2580]: E0804 17:02:29.127295 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43eb054afbc5d61d87dffff5cd2d5cee31f65c6084808d24bd275e6e2b32a044\": container with ID starting with 43eb054afbc5d61d87dffff5cd2d5cee31f65c6084808d24bd275e6e2b32a044 not found: ID does not exist" containerID="43eb054afbc5d61d87dffff5cd2d5cee31f65c6084808d24bd275e6e2b32a044" Aug 04 17:02:29.127371 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.127322 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43eb054afbc5d61d87dffff5cd2d5cee31f65c6084808d24bd275e6e2b32a044"} err="failed to get container status \"43eb054afbc5d61d87dffff5cd2d5cee31f65c6084808d24bd275e6e2b32a044\": rpc error: code = NotFound desc = could not find container \"43eb054afbc5d61d87dffff5cd2d5cee31f65c6084808d24bd275e6e2b32a044\": container with ID starting with 43eb054afbc5d61d87dffff5cd2d5cee31f65c6084808d24bd275e6e2b32a044 not found: ID does not exist" Aug 04 17:02:29.134160 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.134131 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95"] Aug 04 17:02:29.136102 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.136086 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-222eca-79685784f6-nlr95"] Aug 04 17:02:29.145639 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.145622 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz"] Aug 04 17:02:29.149650 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.149631 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-222eca-86969bb5c4-bqxsz"] Aug 04 17:02:29.319295 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.319257 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="052f9387-e170-40be-9a98-37e8cd2f0d09" path="/var/lib/kubelet/pods/052f9387-e170-40be-9a98-37e8cd2f0d09/volumes" Aug 04 17:02:29.319616 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.319600 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15434214-f271-4492-8d0e-71621239d77a" path="/var/lib/kubelet/pods/15434214-f271-4492-8d0e-71621239d77a/volumes" Aug 04 17:02:29.319925 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:29.319910 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f753cb0-606f-4bf6-a064-c5071fc35c61" path="/var/lib/kubelet/pods/4f753cb0-606f-4bf6-a064-c5071fc35c61/volumes" Aug 04 17:02:30.227677 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:30.227644 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rjxvb"] Aug 04 17:02:41.133655 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:41.133620 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-h2b6b_a40745ad-c088-43d9-bf82-2cbf81fb6b09/manager/0.log" Aug 04 17:02:41.620977 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:41.620953 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd47f799b-7qs4j_031e04df-7495-43ec-b8cf-0a5f9d4bcaa9/copy-manifests/0.log" Aug 04 17:02:41.633306 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:41.633269 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd47f799b-7qs4j_031e04df-7495-43ec-b8cf-0a5f9d4bcaa9/manager/0.log" Aug 04 17:02:42.476663 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:42.476639 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29764365-bgb2k_0a9fd05b-a465-465b-b4b8-707e0ea5d4ae/cleanup/0.log" Aug 04 17:02:42.590238 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:42.590209 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29764380-j2kbr_5ad0e5fc-930e-478a-abb7-f2e4b2377239/cleanup/0.log" Aug 04 17:02:44.084609 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:44.084580 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rjxvb_aed77b61-81d6-40d8-b249-d61e28bbe868/limitador/0.log" Aug 04 17:02:44.208003 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:44.207979 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-pbrhj_a0d8a3ab-8c0c-4db0-8a2a-abfc30137224/manager/0.log" Aug 04 17:02:45.462860 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:45.462831 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d8f46644-nmgqm_66928c11-68ad-4888-8ee8-82a21af92095/router/0.log" Aug 04 17:02:46.151815 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:46.151788 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbn5kf_be9906f6-2bce-4957-a0cd-2311ab13d0f7/main/0.log" Aug 04 17:02:46.370258 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:46.370229 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-kj6wb_b776ed5e-e090-40cd-9ccb-d97e80ff878e/main/0.log" Aug 04 17:02:51.714298 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.714246 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wj42w/must-gather-xggjq"] Aug 04 17:02:51.714683 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.714627 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f753cb0-606f-4bf6-a064-c5071fc35c61" containerName="payload-processing" Aug 04 17:02:51.714683 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.714640 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f753cb0-606f-4bf6-a064-c5071fc35c61" containerName="payload-processing" Aug 04 17:02:51.714683 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.714657 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="052f9387-e170-40be-9a98-37e8cd2f0d09" containerName="revoke-keys" Aug 04 17:02:51.714683 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.714662 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="052f9387-e170-40be-9a98-37e8cd2f0d09" containerName="revoke-keys" Aug 04 17:02:51.714683 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.714670 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15434214-f271-4492-8d0e-71621239d77a" containerName="payload-pre-processing" Aug 04 17:02:51.714683 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.714676 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15434214-f271-4492-8d0e-71621239d77a" containerName="payload-pre-processing" Aug 04 17:02:51.714893 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.714743 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f753cb0-606f-4bf6-a064-c5071fc35c61" containerName="payload-processing" Aug 04 17:02:51.714893 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.714751 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="052f9387-e170-40be-9a98-37e8cd2f0d09" containerName="revoke-keys" Aug 04 17:02:51.714893 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.714757 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15434214-f271-4492-8d0e-71621239d77a" containerName="payload-pre-processing" Aug 04 17:02:51.719980 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.719959 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wj42w/must-gather-xggjq" Aug 04 17:02:51.722761 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.722743 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wj42w\"/\"kube-root-ca.crt\"" Aug 04 17:02:51.724099 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.724070 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wj42w\"/\"openshift-service-ca.crt\"" Aug 04 17:02:51.724209 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.724139 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wj42w\"/\"default-dockercfg-ddj78\"" Aug 04 17:02:51.726450 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.726428 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wj42w/must-gather-xggjq"] Aug 04 17:02:51.847213 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.847176 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sfbf\" (UniqueName: \"kubernetes.io/projected/9671150e-9353-4a2c-b02a-b7cb7415da21-kube-api-access-9sfbf\") pod \"must-gather-xggjq\" (UID: \"9671150e-9353-4a2c-b02a-b7cb7415da21\") " pod="openshift-must-gather-wj42w/must-gather-xggjq" Aug 04 17:02:51.847391 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.847245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9671150e-9353-4a2c-b02a-b7cb7415da21-must-gather-output\") pod \"must-gather-xggjq\" (UID: \"9671150e-9353-4a2c-b02a-b7cb7415da21\") " pod="openshift-must-gather-wj42w/must-gather-xggjq" Aug 04 17:02:51.948472 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.948429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9671150e-9353-4a2c-b02a-b7cb7415da21-must-gather-output\") pod \"must-gather-xggjq\" (UID: \"9671150e-9353-4a2c-b02a-b7cb7415da21\") " pod="openshift-must-gather-wj42w/must-gather-xggjq" Aug 04 17:02:51.948605 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.948516 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sfbf\" (UniqueName: \"kubernetes.io/projected/9671150e-9353-4a2c-b02a-b7cb7415da21-kube-api-access-9sfbf\") pod \"must-gather-xggjq\" (UID: \"9671150e-9353-4a2c-b02a-b7cb7415da21\") " pod="openshift-must-gather-wj42w/must-gather-xggjq" Aug 04 17:02:51.948764 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.948745 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9671150e-9353-4a2c-b02a-b7cb7415da21-must-gather-output\") pod \"must-gather-xggjq\" (UID: \"9671150e-9353-4a2c-b02a-b7cb7415da21\") " pod="openshift-must-gather-wj42w/must-gather-xggjq" Aug 04 17:02:51.957157 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:51.957126 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sfbf\" (UniqueName: \"kubernetes.io/projected/9671150e-9353-4a2c-b02a-b7cb7415da21-kube-api-access-9sfbf\") pod \"must-gather-xggjq\" (UID: \"9671150e-9353-4a2c-b02a-b7cb7415da21\") " pod="openshift-must-gather-wj42w/must-gather-xggjq" Aug 04 17:02:52.029980 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:52.029910 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wj42w/must-gather-xggjq" Aug 04 17:02:52.159363 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:52.159335 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wj42w/must-gather-xggjq"] Aug 04 17:02:52.160950 ip-10-0-132-42 kubenswrapper[2580]: W0804 17:02:52.160925 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9671150e_9353_4a2c_b02a_b7cb7415da21.slice/crio-e95b74def64aa72191b09f186492908d4aeae830cd02662508e8acdc0107aa52 WatchSource:0}: Error finding container e95b74def64aa72191b09f186492908d4aeae830cd02662508e8acdc0107aa52: Status 404 returned error can't find the container with id e95b74def64aa72191b09f186492908d4aeae830cd02662508e8acdc0107aa52 Aug 04 17:02:52.225948 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:52.225918 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj42w/must-gather-xggjq" event={"ID":"9671150e-9353-4a2c-b02a-b7cb7415da21","Type":"ContainerStarted","Data":"e95b74def64aa72191b09f186492908d4aeae830cd02662508e8acdc0107aa52"} Aug 04 17:02:53.235339 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:53.234139 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj42w/must-gather-xggjq" event={"ID":"9671150e-9353-4a2c-b02a-b7cb7415da21","Type":"ContainerStarted","Data":"fd01276c9e9bf78feee9d5013782dc97b2840cba25ac8b3065354af9b2e384b5"} Aug 04 17:02:54.243386 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:54.243348 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj42w/must-gather-xggjq" event={"ID":"9671150e-9353-4a2c-b02a-b7cb7415da21","Type":"ContainerStarted","Data":"c7bf8c60b4d575f901adb1649c6fc81fdded215de1279a600599addaf5dfa551"} Aug 04 17:02:54.260936 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:54.260887 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wj42w/must-gather-xggjq" podStartSLOduration=2.385117398 podStartE2EDuration="3.260873316s" totalImagesPullingTime="875.755918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:02:54.259422009 +0000 UTC m=+3675.403754255" watchObservedRunningTime="2026-08-04 17:02:54.260873316 +0000 UTC m=+3675.405205611" Aug 04 17:02:54.609924 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:54.609892 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5hhk7_cb9271e1-f3f4-4afc-b1ce-22d79e60a3db/global-pull-secret-syncer/0.log" Aug 04 17:02:54.744500 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:54.744466 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lnn4g_656bb7e0-4acf-401f-a87a-209fa05614fa/konnectivity-agent/0.log" Aug 04 17:02:54.840876 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:02:54.840847 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-42.ec2.internal_c8a12c8c42b46b94e7355712b1c24093/haproxy/0.log" Aug 04 17:03:00.091897 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:00.091867 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rjxvb_aed77b61-81d6-40d8-b249-d61e28bbe868/limitador/0.log" Aug 04 17:03:00.225471 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:00.225421 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-pbrhj_a0d8a3ab-8c0c-4db0-8a2a-abfc30137224/manager/0.log" Aug 04 17:03:01.805649 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:01.805555 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-rz5dr_84092957-5deb-4cae-b1ba-310f24137f8e/cluster-monitoring-operator/0.log" Aug 04 17:03:02.113701 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:02.113663 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hrf2c_2a8f86c0-08b1-47e6-be77-05099a719954/node-exporter/0.log" Aug 04 17:03:02.133203 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:02.133173 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hrf2c_2a8f86c0-08b1-47e6-be77-05099a719954/kube-rbac-proxy/0.log" Aug 04 17:03:02.153081 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:02.153050 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hrf2c_2a8f86c0-08b1-47e6-be77-05099a719954/init-textfile/0.log" Aug 04 17:03:02.442520 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:02.442446 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-5sk6d_5fecb59b-bad4-4214-92cf-692c79ec448a/prometheus-operator/0.log" Aug 04 17:03:02.461256 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:02.461219 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-5sk6d_5fecb59b-bad4-4214-92cf-692c79ec448a/kube-rbac-proxy/0.log" Aug 04 17:03:02.488048 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:02.488013 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-dvdq8_031a5dd3-4e36-4920-b9a3-d732b8a0a0f5/prometheus-operator-admission-webhook/0.log" Aug 04 17:03:03.936451 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:03.936418 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-tcsrs_6f235ea2-00aa-471c-b792-a2ed764869fa/networking-console-plugin/0.log" Aug 04 17:03:04.301387 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.301270 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x"] Aug 04 17:03:04.306950 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.306920 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.312349 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.312313 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x"] Aug 04 17:03:04.383794 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.383401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89626fec-8d6b-43f3-ad2d-03eb81ca5dd1-lib-modules\") pod \"perf-node-gather-daemonset-ppv9x\" (UID: \"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1\") " pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.383794 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.383457 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/89626fec-8d6b-43f3-ad2d-03eb81ca5dd1-podres\") pod \"perf-node-gather-daemonset-ppv9x\" (UID: \"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1\") " pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.383794 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.383522 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/89626fec-8d6b-43f3-ad2d-03eb81ca5dd1-proc\") pod \"perf-node-gather-daemonset-ppv9x\" (UID: \"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1\") " pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.383794 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.383546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjr4r\" (UniqueName: \"kubernetes.io/projected/89626fec-8d6b-43f3-ad2d-03eb81ca5dd1-kube-api-access-zjr4r\") pod \"perf-node-gather-daemonset-ppv9x\" (UID: \"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1\") " pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.383794 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.383583 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89626fec-8d6b-43f3-ad2d-03eb81ca5dd1-sys\") pod \"perf-node-gather-daemonset-ppv9x\" (UID: \"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1\") " pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.484874 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.484835 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89626fec-8d6b-43f3-ad2d-03eb81ca5dd1-lib-modules\") pod \"perf-node-gather-daemonset-ppv9x\" (UID: \"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1\") " pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.485053 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.484905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/89626fec-8d6b-43f3-ad2d-03eb81ca5dd1-podres\") pod \"perf-node-gather-daemonset-ppv9x\" (UID: \"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1\") " pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.485053 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.484975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/89626fec-8d6b-43f3-ad2d-03eb81ca5dd1-proc\") pod \"perf-node-gather-daemonset-ppv9x\" (UID: \"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1\") " pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.485053 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.485002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjr4r\" (UniqueName: \"kubernetes.io/projected/89626fec-8d6b-43f3-ad2d-03eb81ca5dd1-kube-api-access-zjr4r\") pod \"perf-node-gather-daemonset-ppv9x\" (UID: \"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1\") " pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.485053 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.485038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89626fec-8d6b-43f3-ad2d-03eb81ca5dd1-sys\") pod \"perf-node-gather-daemonset-ppv9x\" (UID: \"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1\") " pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.485270 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.485061 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89626fec-8d6b-43f3-ad2d-03eb81ca5dd1-lib-modules\") pod \"perf-node-gather-daemonset-ppv9x\" (UID: \"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1\") " pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.485270 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.485076 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/89626fec-8d6b-43f3-ad2d-03eb81ca5dd1-podres\") pod \"perf-node-gather-daemonset-ppv9x\" (UID: \"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1\") " pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.485270 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.485131 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/89626fec-8d6b-43f3-ad2d-03eb81ca5dd1-proc\") pod \"perf-node-gather-daemonset-ppv9x\" (UID: \"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1\") " pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.485270 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.485141 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89626fec-8d6b-43f3-ad2d-03eb81ca5dd1-sys\") pod \"perf-node-gather-daemonset-ppv9x\" (UID: \"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1\") " pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.498710 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.498681 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjr4r\" (UniqueName: \"kubernetes.io/projected/89626fec-8d6b-43f3-ad2d-03eb81ca5dd1-kube-api-access-zjr4r\") pod \"perf-node-gather-daemonset-ppv9x\" (UID: \"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1\") " pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.621632 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.621596 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:04.638621 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.638594 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/1.log" Aug 04 17:03:04.645947 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.645919 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-jcbjw_01d084ac-ce47-4d96-bbf0-2e78b83beb17/console-operator/2.log" Aug 04 17:03:04.815643 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:04.815612 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x"] Aug 04 17:03:05.290837 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:05.290754 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-kw7q7_41a9698f-7fff-4a1e-915a-bb1875387c14/download-server/0.log" Aug 04 17:03:05.334455 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:05.334410 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" event={"ID":"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1","Type":"ContainerStarted","Data":"a21088c4646a306b020e79077dadb0c64e211784a7402428954f79225d0f214e"} Aug 04 17:03:05.334655 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:05.334476 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" event={"ID":"89626fec-8d6b-43f3-ad2d-03eb81ca5dd1","Type":"ContainerStarted","Data":"030a443926f55f1a8f7157b52604cc65f55f647227a6ad429ebb1724438fa622"} Aug 04 17:03:05.335362 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:05.335335 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:05.353051 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:05.352995 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" podStartSLOduration=1.352975925 podStartE2EDuration="1.352975925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:03:05.352106797 +0000 UTC m=+3686.496439044" watchObservedRunningTime="2026-08-04 17:03:05.352975925 +0000 UTC m=+3686.497308171" Aug 04 17:03:05.924996 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:05.924966 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-7fvfv_cdaafd1b-8147-4e4d-82c5-578621b9a646/volume-data-source-validator/0.log" Aug 04 17:03:07.011790 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:07.011762 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zj6tj_b557e1a0-4af6-4c29-bb50-214fa9d6882e/dns/0.log" Aug 04 17:03:07.031671 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:07.031644 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zj6tj_b557e1a0-4af6-4c29-bb50-214fa9d6882e/kube-rbac-proxy/0.log" Aug 04 17:03:07.059019 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:07.058995 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2845_add717dd-d023-43b7-bf8b-f50516f053db/dns-node-resolver/0.log" Aug 04 17:03:07.712730 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:07.712691 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vbhgz_56b8e09d-90bb-401d-96a6-20d691168fc3/node-ca/0.log" Aug 04 17:03:09.144600 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:09.144571 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d8f46644-nmgqm_66928c11-68ad-4888-8ee8-82a21af92095/router/0.log" Aug 04 17:03:09.961109 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:09.961081 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v8jrh_a9b9081e-ce41-4a27-97c9-d02d7bd2bab7/serve-healthcheck-canary/0.log" Aug 04 17:03:10.457001 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:10.456970 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-8scs4_8e015152-d986-4b2b-9f93-296d9291b74e/insights-operator/0.log" Aug 04 17:03:10.458211 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:10.458192 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-8scs4_8e015152-d986-4b2b-9f93-296d9291b74e/insights-operator/1.log" Aug 04 17:03:10.478175 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:10.478154 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t7vhm_4081f9ba-ac7b-4f25-b676-4af064e18d99/kube-rbac-proxy/0.log" Aug 04 17:03:10.497219 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:10.497198 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t7vhm_4081f9ba-ac7b-4f25-b676-4af064e18d99/exporter/0.log" Aug 04 17:03:10.515741 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:10.515719 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t7vhm_4081f9ba-ac7b-4f25-b676-4af064e18d99/extractor/0.log" Aug 04 17:03:12.360874 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:12.360449 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wj42w/perf-node-gather-daemonset-ppv9x" Aug 04 17:03:13.014207 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:13.014172 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-h2b6b_a40745ad-c088-43d9-bf82-2cbf81fb6b09/manager/0.log" Aug 04 17:03:13.637255 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:13.636811 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd47f799b-7qs4j_031e04df-7495-43ec-b8cf-0a5f9d4bcaa9/manager/0.log" Aug 04 17:03:13.652992 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:13.652954 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd47f799b-7qs4j_031e04df-7495-43ec-b8cf-0a5f9d4bcaa9/copy-manifests/0.log" Aug 04 17:03:14.433303 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:14.433258 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d8f46644-nmgqm_66928c11-68ad-4888-8ee8-82a21af92095/router/0.log" Aug 04 17:03:15.133230 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:15.133205 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c58867685-x8zbw_e8b1567f-abca-48a2-9b48-0159fa23c5c1/manager/0.log" Aug 04 17:03:16.351693 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:16.351662 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d8f46644-nmgqm_66928c11-68ad-4888-8ee8-82a21af92095/router/0.log" Aug 04 17:03:19.948692 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:19.948656 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-d8g5c_023c82e6-1623-4606-8fad-5eb6984f70f5/migrator/0.log" Aug 04 17:03:19.968648 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:19.968626 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-d8g5c_023c82e6-1623-4606-8fad-5eb6984f70f5/graceful-termination/0.log" Aug 04 17:03:20.323755 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:20.323729 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-v8nwn_cb89b27f-9d1b-41b0-830e-1e464d29d6f1/kube-storage-version-migrator-operator/1.log" Aug 04 17:03:20.324657 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:20.324635 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-v8nwn_cb89b27f-9d1b-41b0-830e-1e464d29d6f1/kube-storage-version-migrator-operator/0.log" Aug 04 17:03:21.307026 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:21.306996 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7q9g5_14b99219-a0ce-4d7b-a976-aa710a8fa6bf/kube-multus/0.log" Aug 04 17:03:21.330671 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:21.330646 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8sdq7_a9c0f306-ee82-4a4a-9680-5b0835964025/kube-multus-additional-cni-plugins/0.log" Aug 04 17:03:21.350709 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:21.350686 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8sdq7_a9c0f306-ee82-4a4a-9680-5b0835964025/egress-router-binary-copy/0.log" Aug 04 17:03:21.370228 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:21.370203 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8sdq7_a9c0f306-ee82-4a4a-9680-5b0835964025/cni-plugins/0.log" Aug 04 17:03:21.389254 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:21.389231 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8sdq7_a9c0f306-ee82-4a4a-9680-5b0835964025/bond-cni-plugin/0.log" Aug 04 17:03:21.407526 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:21.407506 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8sdq7_a9c0f306-ee82-4a4a-9680-5b0835964025/routeoverride-cni/0.log" Aug 04 17:03:21.427460 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:21.427442 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8sdq7_a9c0f306-ee82-4a4a-9680-5b0835964025/whereabouts-cni-bincopy/0.log" Aug 04 17:03:21.446479 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:21.446461 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8sdq7_a9c0f306-ee82-4a4a-9680-5b0835964025/whereabouts-cni/0.log" Aug 04 17:03:21.900391 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:21.900367 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-t79jc_56a1a012-b25b-4c19-b6be-73f382416c22/network-metrics-daemon/0.log" Aug 04 17:03:21.917757 ip-10-0-132-42 kubenswrapper[2580]: I0804 17:03:21.917738 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-t79jc_56a1a012-b25b-4c19-b6be-73f382416c22/kube-rbac-proxy/0.log"