Jul 09 03:56:05.023118 ip-10-0-130-55 systemd[1]: Starting Kubernetes Kubelet... Jul 09 03:56:05.433174 ip-10-0-130-55 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 03:56:05.433174 ip-10-0-130-55 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 03:56:05.433174 ip-10-0-130-55 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 03:56:05.433174 ip-10-0-130-55 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 03:56:05.433174 ip-10-0-130-55 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 03:56:05.433174 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.402145 2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 03:56:05.433174 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407255 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 03:56:05.433174 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407269 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 03:56:05.433174 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407273 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 03:56:05.433174 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407276 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 03:56:05.433174 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407279 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 03:56:05.433174 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407282 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 03:56:05.433174 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407287 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 03:56:05.433174 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407290 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407292 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407295 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407298 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407301 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407304 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407306 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407309 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407312 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407317 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407320 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407324 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407327 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407332 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407336 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407339 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407342 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407345 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407348 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 03:56:05.434144 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407350 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407353 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407356 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407358 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407361 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407365 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407368 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407370 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407373 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407376 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407378 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407382 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407391 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407393 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407396 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407399 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407401 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407404 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407406 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407408 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 03:56:05.434684 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407411 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407414 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407416 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407419 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407421 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407423 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407429 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407433 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407436 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407438 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407441 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407444 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407447 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407449 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407452 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407454 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407457 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407460 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407463 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 03:56:05.435299 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407465 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407468 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407470 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407473 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407475 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407478 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407484 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407486 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407490 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407493 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407495 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407498 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407500 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407503 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407506 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407508 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407510 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407513 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407516 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407518 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 03:56:05.435802 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407520 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407523 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407527 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407530 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407532 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407535 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407537 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407540 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407542 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407545 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407548 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407550 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407553 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.407555 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408327 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408334 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408337 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408340 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408343 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408350 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 03:56:05.436461 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408353 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408357 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408361 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408364 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408367 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408370 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408373 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408376 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408378 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408381 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408384 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408386 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408389 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408392 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408394 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408397 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408400 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408402 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408405 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 03:56:05.437230 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408408 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408410 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408413 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408416 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408418 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408420 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408423 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408425 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408428 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408431 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408433 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408436 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408439 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408447 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408450 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408453 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408456 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408458 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408461 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408464 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 03:56:05.437868 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408466 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408471 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408474 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408477 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408480 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408483 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408485 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408488 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408490 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408493 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408495 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408498 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408501 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408503 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408506 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408509 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408512 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408515 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408518 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 03:56:05.438446 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408520 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408523 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408525 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408528 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408530 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408532 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408535 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408543 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408547 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408551 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408554 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408557 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408560 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408562 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408565 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408569 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408572 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408575 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408577 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 03:56:05.439027 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408580 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408582 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408585 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408587 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408591 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408594 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408597 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408600 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408602 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408604 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408607 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408610 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408613 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408615 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408618 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.408620 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409014 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409019 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409022 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409025 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 03:56:05.439831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409029 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409032 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409035 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409038 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409040 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409043 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409045 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409048 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409050 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409053 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409055 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409058 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409061 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409063 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409066 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409068 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409073 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409076 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409079 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 03:56:05.440419 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409081 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409084 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409086 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409091 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409094 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409097 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409100 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409103 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409106 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409108 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409112 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409115 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409118 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409121 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409124 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409126 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409129 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409132 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409134 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409137 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 03:56:05.441010 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409139 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409141 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409144 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409146 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409149 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409151 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409154 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409157 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409159 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409163 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409166 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409168 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409171 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409173 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409177 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409181 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409184 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409187 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409189 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409192 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 03:56:05.441580 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409194 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409196 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409199 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409202 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409205 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409208 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409212 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409215 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409218 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409221 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409223 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409226 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409228 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409231 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409233 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409236 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409238 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409241 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409243 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409245 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 03:56:05.442192 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409248 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409252 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409255 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409259 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409261 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409264 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409266 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409269 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409271 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409273 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409276 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409278 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409281 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409284 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409286 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.409288 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409380 2571 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409395 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409402 2571 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409406 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409410 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 03:56:05.442749 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409414 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409418 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409422 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409425 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409428 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409432 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409435 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409438 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409441 2571 flags.go:64] FLAG: --cgroup-root="" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409444 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409447 2571 flags.go:64] FLAG: --client-ca-file="" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409450 2571 flags.go:64] FLAG: --cloud-provider="external" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409454 2571 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409459 2571 flags.go:64] FLAG: --cluster-domain="" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409463 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409466 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409469 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409473 2571 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409476 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409479 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409482 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409486 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409488 2571 flags.go:64] FLAG: --contention-profiling="false" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409492 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 03:56:05.596113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409495 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 03:56:05.489556 ip-10-0-130-55 systemd[1]: Started Kubernetes Kubelet. Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409498 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409501 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409506 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409509 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409513 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409516 2571 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409519 2571 flags.go:64] FLAG: --enable-server="true" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409523 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409528 2571 flags.go:64] FLAG: --event-burst="100" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409531 2571 flags.go:64] FLAG: --event-qps="50" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409534 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409537 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409540 2571 flags.go:64] FLAG: --eviction-hard="" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409543 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409546 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409549 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409552 2571 flags.go:64] FLAG: --eviction-soft="" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409555 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409558 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409562 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409565 2571 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409570 2571 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409573 2571 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409575 2571 flags.go:64] FLAG: --feature-gates="" Jul 09 03:56:05.599702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409579 2571 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409582 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409585 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409588 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409592 2571 flags.go:64] FLAG: --healthz-port="10248" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409595 2571 flags.go:64] FLAG: --help="false" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409597 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-130-55.ec2.internal" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409600 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409603 2571 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409606 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409609 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409613 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409615 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409618 2571 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409621 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409624 2571 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409627 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409630 2571 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409633 2571 flags.go:64] FLAG: --kube-reserved="" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409636 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409639 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409641 2571 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409644 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409647 2571 flags.go:64] FLAG: --lock-file="" Jul 09 03:56:05.617628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409649 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409652 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409655 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409663 2571 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409666 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409671 2571 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409674 2571 flags.go:64] FLAG: --logging-format="text" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409677 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409680 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409683 2571 flags.go:64] FLAG: --manifest-url="" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409686 2571 flags.go:64] FLAG: --manifest-url-header="" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409690 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409693 2571 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409697 2571 flags.go:64] FLAG: --max-pods="110" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409699 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409702 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409705 2571 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409708 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409711 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409714 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409717 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409724 2571 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409727 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409730 2571 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 03:56:05.618976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409733 2571 flags.go:64] FLAG: --pod-cidr="" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409736 2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409742 2571 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409745 2571 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409748 2571 flags.go:64] FLAG: --pods-per-core="0" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409751 2571 flags.go:64] FLAG: --port="10250" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409754 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409757 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d505b23c42e18b70" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409760 2571 flags.go:64] FLAG: --qos-reserved="" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409764 2571 flags.go:64] FLAG: --read-only-port="10255" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409766 2571 flags.go:64] FLAG: --register-node="true" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409769 2571 flags.go:64] FLAG: --register-with-taints="" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409774 2571 flags.go:64] FLAG: --registry-burst="10" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409779 2571 flags.go:64] FLAG: --registry-qps="5" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409781 2571 flags.go:64] FLAG: --reserved-cpus="" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409784 2571 flags.go:64] FLAG: --reserved-memory="" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409788 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409791 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409794 2571 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409797 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409800 2571 flags.go:64] FLAG: --runonce="false" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409803 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409806 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409809 2571 flags.go:64] FLAG: --seccomp-default="false" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409811 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 03:56:05.621411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409814 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409817 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409820 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409823 2571 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409826 2571 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409829 2571 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409832 2571 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409834 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409837 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409841 2571 flags.go:64] FLAG: --system-cgroups="" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409843 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409849 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409864 2571 flags.go:64] FLAG: --tls-cert-file="" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409867 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409871 2571 flags.go:64] FLAG: --tls-min-version="" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409874 2571 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409876 2571 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409879 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409882 2571 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409887 2571 flags.go:64] FLAG: --v="2" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409892 2571 flags.go:64] FLAG: --version="false" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409896 2571 flags.go:64] FLAG: --vmodule="" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409899 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.409903 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410023 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 03:56:05.622491 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410027 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410030 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410033 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410036 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410039 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410043 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410047 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410051 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410054 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410057 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410060 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410062 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410065 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410068 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410070 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410073 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410075 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410078 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410081 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 03:56:05.623503 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410083 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410086 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410089 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410091 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410094 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410096 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410099 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410103 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410108 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410110 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410113 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410116 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410118 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410121 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410123 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410126 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410129 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410131 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410134 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410137 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 03:56:05.624237 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410139 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410142 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410144 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410147 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410149 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410152 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410154 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410157 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410159 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410162 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410164 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410167 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410170 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410172 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410175 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410177 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410179 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410182 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410184 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410188 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 03:56:05.624964 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410192 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410195 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410197 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410200 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410202 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410205 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410208 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410210 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410213 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410215 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410218 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410220 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410223 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410225 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410228 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410230 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410233 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410236 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410238 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410241 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 03:56:05.625666 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410243 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410246 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410250 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410253 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410256 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410259 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410262 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410264 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410267 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410269 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410272 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410275 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410279 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410282 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410284 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410287 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410290 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410293 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 09 03:56:05.626398 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.410295 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 03:56:05.627033 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.410983 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 03:56:05.627033 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.418047 2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 09 03:56:05.627033 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.418060 2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 03:56:05.627033 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.418077 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 03:56:05.627033 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.418083 2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 03:56:05.627033 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418144 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 03:56:05.627033 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418149 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 03:56:05.627033 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418152 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 03:56:05.627033 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418155 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 03:56:05.627033 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418158 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 03:56:05.627033 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418162 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 03:56:05.627033 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418165 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 03:56:05.627033 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418168 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 03:56:05.627033 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418171 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 03:56:05.627033 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418174 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418177 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418180 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418189 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418192 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418194 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418197 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418200 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418203 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418205 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418208 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418214 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418217 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418220 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418223 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418226 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418228 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418231 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418233 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418236 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 03:56:05.627625 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418239 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418242 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418244 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418247 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418249 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418252 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418254 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418257 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418259 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418269 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418273 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418276 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418279 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418281 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418284 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418286 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418294 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418297 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418300 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 03:56:05.628276 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418302 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418305 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418307 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418310 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418315 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418318 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418321 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418323 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418326 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418329 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418331 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418334 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418337 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418339 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418342 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418344 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418347 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418349 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418352 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418354 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 03:56:05.628881 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418357 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418360 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418363 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418365 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418368 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418370 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418373 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418375 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418378 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418380 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418388 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418391 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418393 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418396 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418399 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418401 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418406 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418410 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418413 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418416 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 03:56:05.629529 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418419 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 03:56:05.630257 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418422 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 03:56:05.630257 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418424 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 03:56:05.630257 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418427 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 03:56:05.630257 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418429 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 03:56:05.630257 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418432 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 03:56:05.630257 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418435 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 03:56:05.630257 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418439 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 03:56:05.630257 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418442 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 03:56:05.630257 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418445 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 03:56:05.630257 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418447 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 03:56:05.630257 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.418452 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 03:56:05.630257 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418612 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 03:56:05.630257 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418616 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 03:56:05.630257 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418619 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418622 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418624 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418627 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418629 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418635 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418638 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418640 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418643 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418651 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418653 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418656 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418658 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418661 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418666 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418671 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418673 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418676 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418679 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 03:56:05.630722 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418681 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418685 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418689 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418691 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418694 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418697 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418699 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418702 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418704 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418707 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418709 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418712 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418714 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418717 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418719 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418722 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418724 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418727 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418729 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 03:56:05.631327 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418732 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418734 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418737 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418739 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418747 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418750 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418752 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418755 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418758 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418763 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418766 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418768 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418771 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418774 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418776 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418779 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418781 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418784 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418786 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 03:56:05.632231 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418789 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418792 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418796 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418799 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418801 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418804 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418807 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418810 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418812 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418815 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418818 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418820 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418822 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418825 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418827 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418830 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418832 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418834 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418842 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418845 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 03:56:05.632885 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418847 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418869 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418874 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418876 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418879 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418881 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418884 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418886 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418889 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418891 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418894 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418897 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418899 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418903 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418906 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418909 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418912 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418914 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418917 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 03:56:05.633539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:05.418920 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 03:56:05.634200 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.418925 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 03:56:05.634200 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.419017 2571 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 03:56:05.634200 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.422933 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 03:56:05.634200 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.424314 2571 server.go:1019] "Starting client certificate rotation" Jul 09 03:56:05.634200 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.424419 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 03:56:05.634200 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.424453 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 03:56:05.634200 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.433572 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 03:56:05.634200 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.434836 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 03:56:05.634200 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.446456 2571 log.go:25] "Validated CRI v1 runtime API" Jul 09 03:56:05.634200 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.448719 2571 log.go:25] "Validated CRI v1 image API" Jul 09 03:56:05.634200 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.449585 2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 03:56:05.634200 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.454983 2571 fs.go:135] Filesystem UUIDs: map[65766035-51df-4b9b-acf9-07f388d4be33:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ecbe6d3f-dddb-4551-be9c-3fcbb5389883:/dev/nvme0n1p3] Jul 09 03:56:05.634633 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.454998 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 03:56:05.634857 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.460271 2571 manager.go:216] Machine: {Timestamp:2026-07-09 03:56:05.459062741 +0000 UTC m=+0.317933832 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101783 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d702dfa4799441d8134b1edba25fa SystemUUID:ec2d702d-fa47-9944-1d81-34b1edba25fa BootID:093c6783-cc61-48da-8e67-77938cd2cfb8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2c:59:8c:bd:1d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2c:59:8c:bd:1d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:5b:f6:5b:11:ab Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 03:56:05.634857 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.460382 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 03:56:05.634857 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.460458 2571 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 03:56:05.634857 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.461754 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 03:56:05.634857 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.463554 2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 03:56:05.634857 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.463580 2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-55.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 03:56:05.634857 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.464359 2571 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.464371 2571 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.464379 2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.465668 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.467384 2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.470748 2571 state_mem.go:36] "Initialized new in-memory state store" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.470849 2571 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.470891 2571 kubelet.go:486] "Attempting to sync node with API server" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.470908 2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.470921 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.470930 2571 kubelet.go:398] "Adding apiserver pod source" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.470941 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.471250 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.471265 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.474270 2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.482978 2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.483007 2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.484023 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-55.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.484024 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.484235 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.484255 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.484264 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.484272 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.484284 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.484293 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.484301 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.484306 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.484312 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.484318 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.484327 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.484332 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.485788 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.485799 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.488770 2571 server.go:1268] "Started kubelet" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.488877 2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.488876 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.488996 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.490153 2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.490735 2571 server.go:310] "Adding debug handlers to kubelet server" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.494829 2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 03:56:05.635293 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.498533 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-55.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.498553 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-55.ec2.internal.18c082e35e39b98f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-55.ec2.internal,UID:ip-10-0-130-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-55.ec2.internal,},FirstTimestamp:2026-07-09 03:56:05.488744847 +0000 UTC m=+0.347615938,LastTimestamp:2026-07-09 03:56:05.488744847 +0000 UTC m=+0.347615938,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-55.ec2.internal,}" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.502702 2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.502712 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.502785 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.503244 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.503546 2571 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.503563 2571 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.503697 2571 reconstruct.go:97] "Volume reconstruction finished" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.503732 2571 reconciler.go:29] "Reconciler: start to sync state" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.504222 2571 factory.go:55] Registering systemd factory Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.504234 2571 factory.go:223] Registration of the systemd container factory successfully Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.504445 2571 factory.go:153] Registering CRI-O factory Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.504461 2571 factory.go:223] Registration of the crio container factory successfully Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.504513 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.504536 2571 factory.go:103] Registering Raw factory Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.504550 2571 manager.go:1196] Started watching for new ooms in manager Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.504926 2571 manager.go:349] Starting recovery of all containers Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.506831 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nsb97" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.516031 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.516471 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-55.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.516831 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nsb97" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.522349 2571 manager.go:354] Recovery completed Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.526225 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.528542 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientMemory" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.528564 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.528574 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientPID" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.529080 2571 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.529088 2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.529097 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.529101 2571 state_mem.go:36] "Initialized new in-memory state store" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.529785 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientMemory" Jul 09 03:56:05.636764 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.529804 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.529814 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientPID" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.530194 2571 policy_none.go:49] "None policy: Start" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.530207 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.530217 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.530381 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-55.ec2.internal.18c082e360992213 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-55.ec2.internal,UID:ip-10-0-130-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-55.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-55.ec2.internal,},FirstTimestamp:2026-07-09 03:56:05.528551955 +0000 UTC m=+0.387423046,LastTimestamp:2026-07-09 03:56:05.528551955 +0000 UTC m=+0.387423046,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-55.ec2.internal,}" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.533715 2571 policy_none.go:47] "Start" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.581486 2571 manager.go:337] "Starting Device Plugin manager" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.581515 2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.581525 2571 server.go:87] "Starting device plugin registration server" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.581770 2571 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.581784 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.581863 2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.581947 2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.581956 2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.582481 2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.582524 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-55.ec2.internal\" not found" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.618025 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.619574 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.619612 2571 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.619647 2571 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.619686 2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 03:56:05.637714 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.622615 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.686672 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.687686 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientMemory" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.687716 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.687728 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientPID" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.687756 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-55.ec2.internal" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.698416 2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-55.ec2.internal" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.698439 2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-55.ec2.internal\": node \"ip-10-0-130-55.ec2.internal\" not found" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.720835 2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal"] Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.720900 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.722165 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientMemory" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.722188 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.722199 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientPID" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.723450 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.723589 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.723617 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.724302 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientMemory" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.724321 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.724331 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientPID" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.724522 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientMemory" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.724534 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.724542 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientPID" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.725078 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.725978 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.726001 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.726759 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientMemory" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.726778 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.726787 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientPID" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.747560 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-55.ec2.internal\" not found" node="ip-10-0-130-55.ec2.internal" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.752156 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-55.ec2.internal\" not found" node="ip-10-0-130-55.ec2.internal" Jul 09 03:56:05.855988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.806262 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/77031bd827de1cac5285b54b00e4a311-config\") pod \"kube-apiserver-proxy-ip-10-0-130-55.ec2.internal\" (UID: \"77031bd827de1cac5285b54b00e4a311\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" Jul 09 03:56:05.857102 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.806285 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3abb2fe191c6b50ca1da05d67be83670-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal\" (UID: \"3abb2fe191c6b50ca1da05d67be83670\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 09 03:56:05.857102 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.806314 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3abb2fe191c6b50ca1da05d67be83670-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal\" (UID: \"3abb2fe191c6b50ca1da05d67be83670\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 09 03:56:05.857102 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.825151 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 09 03:56:05.936630 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.906838 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/77031bd827de1cac5285b54b00e4a311-config\") pod \"kube-apiserver-proxy-ip-10-0-130-55.ec2.internal\" (UID: \"77031bd827de1cac5285b54b00e4a311\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" Jul 09 03:56:05.936630 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.906894 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3abb2fe191c6b50ca1da05d67be83670-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal\" (UID: \"3abb2fe191c6b50ca1da05d67be83670\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 09 03:56:05.936630 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.906912 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3abb2fe191c6b50ca1da05d67be83670-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal\" (UID: \"3abb2fe191c6b50ca1da05d67be83670\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 09 03:56:05.936630 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.906941 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3abb2fe191c6b50ca1da05d67be83670-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal\" (UID: \"3abb2fe191c6b50ca1da05d67be83670\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 09 03:56:05.936630 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.906944 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3abb2fe191c6b50ca1da05d67be83670-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal\" (UID: \"3abb2fe191c6b50ca1da05d67be83670\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 09 03:56:05.936630 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:05.906944 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/77031bd827de1cac5285b54b00e4a311-config\") pod \"kube-apiserver-proxy-ip-10-0-130-55.ec2.internal\" (UID: \"77031bd827de1cac5285b54b00e4a311\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" Jul 09 03:56:05.936630 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:05.926379 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 09 03:56:06.109688 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:06.027959 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 09 03:56:06.109688 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:06.051918 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" Jul 09 03:56:06.109688 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:06.056722 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 09 03:56:06.345546 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:06.128570 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 09 03:56:06.345546 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:06.229358 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 09 03:56:06.345546 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:06.330365 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 09 03:56:06.516327 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:06.424087 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 03:56:06.516327 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:06.424223 2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 03:56:06.516327 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:06.430516 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 09 03:56:06.522040 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:06.521495 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 03:51:06 +0000 UTC" deadline="2028-02-14 09:20:05.147486961 +0000 UTC" Jul 09 03:56:06.522040 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:06.521522 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14045h23m58.625968674s" Jul 09 03:56:06.531683 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:06.531656 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 09 03:56:06.631910 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:06.631883 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 09 03:56:06.712340 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:06.712191 2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 03:56:06.795812 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:06.795779 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77031bd827de1cac5285b54b00e4a311.slice/crio-fb514f5e52a33ec9edf5daec06acba2c69a4e5d00740b252edb487861f668b85 WatchSource:0}: Error finding container fb514f5e52a33ec9edf5daec06acba2c69a4e5d00740b252edb487861f668b85: Status 404 returned error can't find the container with id fb514f5e52a33ec9edf5daec06acba2c69a4e5d00740b252edb487861f668b85 Jul 09 03:56:06.796297 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:06.796273 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abb2fe191c6b50ca1da05d67be83670.slice/crio-009af49f81a398d5261f950f118a77616f145583a08f7e1b582311bcec62f07a WatchSource:0}: Error finding container 009af49f81a398d5261f950f118a77616f145583a08f7e1b582311bcec62f07a: Status 404 returned error can't find the container with id 009af49f81a398d5261f950f118a77616f145583a08f7e1b582311bcec62f07a Jul 09 03:56:06.801246 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:06.801227 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 03:56:06.802950 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:06.802926 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" Jul 09 03:56:06.813454 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:06.813423 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 03:56:06.815332 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:06.815315 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 09 03:56:06.827269 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:06.827253 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 03:56:06.943851 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:06.943825 2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 03:56:07.009246 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.009224 2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 03:56:07.472073 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.472030 2571 apiserver.go:52] "Watching apiserver" Jul 09 03:56:07.478828 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.478801 2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 03:56:07.480192 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.480161 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-mwnbk","kube-system/global-pull-secret-syncer-28sj8","kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal","openshift-cluster-node-tuning-operator/tuned-62cz2","openshift-dns/node-resolver-ltzfl","openshift-image-registry/node-ca-gn78p","openshift-multus/multus-59wnp","openshift-multus/multus-additional-cni-plugins-bb7z2","openshift-multus/network-metrics-daemon-fvb66","kube-system/konnectivity-agent-vf56w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal","openshift-network-diagnostics/network-check-target-c7ws6","openshift-network-operator/iptables-alerter-kc98t"] Jul 09 03:56:07.481685 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.481666 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.482925 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.482902 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:07.483009 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:07.482977 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:07.484005 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.483983 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 03:56:07.484125 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.484003 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 03:56:07.484125 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.484045 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hkj5r\"" Jul 09 03:56:07.484308 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.484265 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kc98t" Jul 09 03:56:07.484375 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.484343 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 03:56:07.484718 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.484702 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 03:56:07.485432 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.485409 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.486658 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.486636 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 03:56:07.486658 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.486649 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ltzfl" Jul 09 03:56:07.486823 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.486678 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 03:56:07.486823 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.486815 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 03:56:07.486930 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.486656 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-76lgp\"" Jul 09 03:56:07.487562 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.487542 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 03:56:07.487704 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.487612 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 03:56:07.487856 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.487833 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dhw4m\"" Jul 09 03:56:07.488694 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.488668 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gn78p" Jul 09 03:56:07.489162 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.488908 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 03:56:07.489330 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.489312 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ch7vv\"" Jul 09 03:56:07.489525 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.489494 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 03:56:07.489899 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.489866 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:07.490002 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:07.489957 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:07.490952 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.490933 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 03:56:07.491299 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.491280 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gpfrx\"" Jul 09 03:56:07.491396 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.491381 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 03:56:07.491489 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.491475 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 03:56:07.491745 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.491731 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.493234 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.492891 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:07.493234 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:07.492953 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:07.493860 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.493841 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 03:56:07.494684 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.494151 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 03:56:07.494684 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.494212 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 03:56:07.494684 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.494243 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 03:56:07.494684 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.494353 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 03:56:07.494684 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.494426 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 03:56:07.494684 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.494477 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vf56w" Jul 09 03:56:07.494684 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.494543 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cszfc\"" Jul 09 03:56:07.495809 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.495790 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.497018 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.496877 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 03:56:07.497018 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.496913 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 03:56:07.497018 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.496953 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lqrf9\"" Jul 09 03:56:07.497228 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.497163 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.498496 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.498473 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 03:56:07.498600 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.498517 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 03:56:07.501562 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.499223 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 03:56:07.501562 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.499222 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cljjc\"" Jul 09 03:56:07.501562 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.499773 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 03:56:07.501562 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.499897 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l9k5g\"" Jul 09 03:56:07.501562 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.500182 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 03:56:07.504178 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.503893 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 03:56:07.504870 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.504853 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 03:56:07.513666 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.513637 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07737c3c-20ee-414b-8dde-b9661df9e485-host-slash\") pod \"iptables-alerter-kc98t\" (UID: \"07737c3c-20ee-414b-8dde-b9661df9e485\") " pod="openshift-network-operator/iptables-alerter-kc98t" Jul 09 03:56:07.513784 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.513681 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-slash\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.513784 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.513709 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.513784 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.513735 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-system-cni-dir\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.513784 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.513762 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-device-dir\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.514023 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.513788 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/747cde48-17af-4cda-ab13-0a621520741c-cni-binary-copy\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.514023 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.513818 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/747cde48-17af-4cda-ab13-0a621520741c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.514023 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.513863 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-run-k8s-cni-cncf-io\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.514023 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.513937 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-hostroot\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.514023 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.513972 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-run-multus-certs\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.514276 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514034 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-kubernetes\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.514276 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514058 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e23a13f6-edd0-42f2-a03e-ba0b1cf7186d-hosts-file\") pod \"node-resolver-ltzfl\" (UID: \"e23a13f6-edd0-42f2-a03e-ba0b1cf7186d\") " pod="openshift-dns/node-resolver-ltzfl" Jul 09 03:56:07.514276 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514084 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmrj4\" (UniqueName: \"kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4\") pod \"network-check-target-c7ws6\" (UID: \"087e5e53-db4d-4eed-bad5-fb6949ff07c3\") " pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:07.514276 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514110 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-registration-dir\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.514276 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514133 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-os-release\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.514276 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514156 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-modprobe-d\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.514276 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514214 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-var-lib-openvswitch\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.514276 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514252 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-systemd\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.514660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514281 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-systemd-units\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.514660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514311 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-run-ovn\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.514660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514334 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d4983fe-8ee1-40b3-89f6-3508333126b3-dbus\") pod \"global-pull-secret-syncer-28sj8\" (UID: \"0d4983fe-8ee1-40b3-89f6-3508333126b3\") " pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:07.514660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514358 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.514660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514383 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-socket-dir\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.514660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514408 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-var-lib-cni-bin\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.514660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514432 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2445180-4ff5-44bb-8e74-1b581138eeca-tmp\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.514660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514458 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs\") pod \"network-metrics-daemon-fvb66\" (UID: \"579aea83-7020-4eb2-9cc9-eb6318890a3e\") " pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:07.514660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514484 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-kubelet\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.514660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514530 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3cd86c1a-eab6-4284-9182-8c034842eaf7-env-overrides\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.514660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514565 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3cd86c1a-eab6-4284-9182-8c034842eaf7-ovn-node-metrics-cert\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.514660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514622 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3cd86c1a-eab6-4284-9182-8c034842eaf7-ovnkube-script-lib\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.514660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514650 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt7w6\" (UniqueName: \"kubernetes.io/projected/650443f7-bb0e-4c54-b0bb-996c52116d90-kube-api-access-tt7w6\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.515382 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514674 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-cnibin\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.515382 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514703 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2217f5b-d7af-42e3-b058-b2e3570e8156-cni-binary-copy\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.515382 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514729 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d4983fe-8ee1-40b3-89f6-3508333126b3-kubelet-config\") pod \"global-pull-secret-syncer-28sj8\" (UID: \"0d4983fe-8ee1-40b3-89f6-3508333126b3\") " pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:07.515382 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514753 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54h2c\" (UniqueName: \"kubernetes.io/projected/07737c3c-20ee-414b-8dde-b9661df9e485-kube-api-access-54h2c\") pod \"iptables-alerter-kc98t\" (UID: \"07737c3c-20ee-414b-8dde-b9661df9e485\") " pod="openshift-network-operator/iptables-alerter-kc98t" Jul 09 03:56:07.515382 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514776 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-run-netns\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.515382 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514814 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-multus-cni-dir\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.515382 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514874 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-var-lib-kubelet\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.515382 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514901 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15fb2067-4603-4c89-95d2-92f7f719166a-host\") pod \"node-ca-gn78p\" (UID: \"15fb2067-4603-4c89-95d2-92f7f719166a\") " pod="openshift-image-registry/node-ca-gn78p" Jul 09 03:56:07.515382 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514940 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/15fb2067-4603-4c89-95d2-92f7f719166a-serviceca\") pod \"node-ca-gn78p\" (UID: \"15fb2067-4603-4c89-95d2-92f7f719166a\") " pod="openshift-image-registry/node-ca-gn78p" Jul 09 03:56:07.515382 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.514967 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/388dfe09-6614-4173-a31b-65185f254dc4-agent-certs\") pod \"konnectivity-agent-vf56w\" (UID: \"388dfe09-6614-4173-a31b-65185f254dc4\") " pod="kube-system/konnectivity-agent-vf56w" Jul 09 03:56:07.515382 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515003 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-etc-openvswitch\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.515382 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515033 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-cni-netd\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.515382 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515073 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2217f5b-d7af-42e3-b058-b2e3570e8156-multus-daemon-config\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.515382 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515123 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/747cde48-17af-4cda-ab13-0a621520741c-system-cni-dir\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.516074 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515153 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/747cde48-17af-4cda-ab13-0a621520741c-cnibin\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.516074 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515183 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/747cde48-17af-4cda-ab13-0a621520741c-os-release\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.516074 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515221 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07737c3c-20ee-414b-8dde-b9661df9e485-iptables-alerter-script\") pod \"iptables-alerter-kc98t\" (UID: \"07737c3c-20ee-414b-8dde-b9661df9e485\") " pod="openshift-network-operator/iptables-alerter-kc98t" Jul 09 03:56:07.516074 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515273 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-run-systemd\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.516074 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515306 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-log-socket\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.516074 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515338 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3cd86c1a-eab6-4284-9182-8c034842eaf7-ovnkube-config\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.516074 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515371 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-etc-selinux\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.516074 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515380 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 03:56:07.516074 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515396 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-var-lib-kubelet\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.516074 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515427 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl5ct\" (UniqueName: \"kubernetes.io/projected/a2217f5b-d7af-42e3-b058-b2e3570e8156-kube-api-access-zl5ct\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.516074 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515450 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-lib-modules\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.516074 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515475 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-multus-socket-dir-parent\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.516074 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515498 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-sysctl-conf\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.516074 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515529 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-sys\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.517065 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515557 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-tuned\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.517065 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515586 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrjxg\" (UniqueName: \"kubernetes.io/projected/e2445180-4ff5-44bb-8e74-1b581138eeca-kube-api-access-qrjxg\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.517065 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515614 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23a13f6-edd0-42f2-a03e-ba0b1cf7186d-tmp-dir\") pod \"node-resolver-ltzfl\" (UID: \"e23a13f6-edd0-42f2-a03e-ba0b1cf7186d\") " pod="openshift-dns/node-resolver-ltzfl" Jul 09 03:56:07.517065 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515639 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/388dfe09-6614-4173-a31b-65185f254dc4-konnectivity-ca\") pod \"konnectivity-agent-vf56w\" (UID: \"388dfe09-6614-4173-a31b-65185f254dc4\") " pod="kube-system/konnectivity-agent-vf56w" Jul 09 03:56:07.517065 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515669 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pc86\" (UniqueName: \"kubernetes.io/projected/747cde48-17af-4cda-ab13-0a621520741c-kube-api-access-9pc86\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.517065 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515694 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms2gs\" (UniqueName: \"kubernetes.io/projected/3cd86c1a-eab6-4284-9182-8c034842eaf7-kube-api-access-ms2gs\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.517065 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515719 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-run-openvswitch\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.517065 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515743 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-node-log\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.517065 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515770 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-run-netns\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.517065 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515793 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-sysconfig\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.517065 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515827 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-sysctl-d\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.517065 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.515924 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvhz4\" (UniqueName: \"kubernetes.io/projected/15fb2067-4603-4c89-95d2-92f7f719166a-kube-api-access-zvhz4\") pod \"node-ca-gn78p\" (UID: \"15fb2067-4603-4c89-95d2-92f7f719166a\") " pod="openshift-image-registry/node-ca-gn78p" Jul 09 03:56:07.517065 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.516000 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-run-ovn-kubernetes\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.517065 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.516027 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret\") pod \"global-pull-secret-syncer-28sj8\" (UID: \"0d4983fe-8ee1-40b3-89f6-3508333126b3\") " pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:07.517668 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.516076 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-sys-fs\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.517668 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.516100 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-etc-kubernetes\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.517668 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.516172 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-host\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.517668 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.516197 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-cni-bin\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.517668 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.516214 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-var-lib-cni-multus\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.517668 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.516230 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-run\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.517668 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.516260 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdsm6\" (UniqueName: \"kubernetes.io/projected/e23a13f6-edd0-42f2-a03e-ba0b1cf7186d-kube-api-access-tdsm6\") pod \"node-resolver-ltzfl\" (UID: \"e23a13f6-edd0-42f2-a03e-ba0b1cf7186d\") " pod="openshift-dns/node-resolver-ltzfl" Jul 09 03:56:07.517668 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.516297 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/747cde48-17af-4cda-ab13-0a621520741c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.517668 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.516322 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/747cde48-17af-4cda-ab13-0a621520741c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.517668 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.516378 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-multus-conf-dir\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.517668 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.516396 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr2f5\" (UniqueName: \"kubernetes.io/projected/579aea83-7020-4eb2-9cc9-eb6318890a3e-kube-api-access-hr2f5\") pod \"network-metrics-daemon-fvb66\" (UID: \"579aea83-7020-4eb2-9cc9-eb6318890a3e\") " pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:07.543386 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.543360 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pp96s" Jul 09 03:56:07.551802 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.551781 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pp96s" Jul 09 03:56:07.592022 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.591991 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 03:56:07.617298 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617267 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e23a13f6-edd0-42f2-a03e-ba0b1cf7186d-hosts-file\") pod \"node-resolver-ltzfl\" (UID: \"e23a13f6-edd0-42f2-a03e-ba0b1cf7186d\") " pod="openshift-dns/node-resolver-ltzfl" Jul 09 03:56:07.617532 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617313 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrj4\" (UniqueName: \"kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4\") pod \"network-check-target-c7ws6\" (UID: \"087e5e53-db4d-4eed-bad5-fb6949ff07c3\") " pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:07.617532 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617483 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e23a13f6-edd0-42f2-a03e-ba0b1cf7186d-hosts-file\") pod \"node-resolver-ltzfl\" (UID: \"e23a13f6-edd0-42f2-a03e-ba0b1cf7186d\") " pod="openshift-dns/node-resolver-ltzfl" Jul 09 03:56:07.617659 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617531 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-registration-dir\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.617659 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617560 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-os-release\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.617659 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617606 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-modprobe-d\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.617659 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617634 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-var-lib-openvswitch\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.617865 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617664 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-systemd\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.617865 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617689 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-systemd-units\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.617865 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617713 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-run-ovn\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.617865 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617764 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d4983fe-8ee1-40b3-89f6-3508333126b3-dbus\") pod \"global-pull-secret-syncer-28sj8\" (UID: \"0d4983fe-8ee1-40b3-89f6-3508333126b3\") " pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:07.617865 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617790 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.617865 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617814 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-socket-dir\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.617865 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617862 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-var-lib-cni-bin\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.618243 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617891 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2445180-4ff5-44bb-8e74-1b581138eeca-tmp\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.618243 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617932 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs\") pod \"network-metrics-daemon-fvb66\" (UID: \"579aea83-7020-4eb2-9cc9-eb6318890a3e\") " pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:07.618243 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617959 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-kubelet\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.618243 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.617996 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3cd86c1a-eab6-4284-9182-8c034842eaf7-env-overrides\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.618243 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618021 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3cd86c1a-eab6-4284-9182-8c034842eaf7-ovn-node-metrics-cert\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.618243 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618048 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3cd86c1a-eab6-4284-9182-8c034842eaf7-ovnkube-script-lib\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.618243 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618074 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt7w6\" (UniqueName: \"kubernetes.io/projected/650443f7-bb0e-4c54-b0bb-996c52116d90-kube-api-access-tt7w6\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.618243 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618099 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-cnibin\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.618243 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618123 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2217f5b-d7af-42e3-b058-b2e3570e8156-cni-binary-copy\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.618243 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618149 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d4983fe-8ee1-40b3-89f6-3508333126b3-kubelet-config\") pod \"global-pull-secret-syncer-28sj8\" (UID: \"0d4983fe-8ee1-40b3-89f6-3508333126b3\") " pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:07.618243 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618173 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54h2c\" (UniqueName: \"kubernetes.io/projected/07737c3c-20ee-414b-8dde-b9661df9e485-kube-api-access-54h2c\") pod \"iptables-alerter-kc98t\" (UID: \"07737c3c-20ee-414b-8dde-b9661df9e485\") " pod="openshift-network-operator/iptables-alerter-kc98t" Jul 09 03:56:07.618243 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618196 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-run-netns\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.618243 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618222 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-multus-cni-dir\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.618243 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618245 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-var-lib-kubelet\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.618960 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618267 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15fb2067-4603-4c89-95d2-92f7f719166a-host\") pod \"node-ca-gn78p\" (UID: \"15fb2067-4603-4c89-95d2-92f7f719166a\") " pod="openshift-image-registry/node-ca-gn78p" Jul 09 03:56:07.618960 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618291 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/15fb2067-4603-4c89-95d2-92f7f719166a-serviceca\") pod \"node-ca-gn78p\" (UID: \"15fb2067-4603-4c89-95d2-92f7f719166a\") " pod="openshift-image-registry/node-ca-gn78p" Jul 09 03:56:07.618960 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618315 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/388dfe09-6614-4173-a31b-65185f254dc4-agent-certs\") pod \"konnectivity-agent-vf56w\" (UID: \"388dfe09-6614-4173-a31b-65185f254dc4\") " pod="kube-system/konnectivity-agent-vf56w" Jul 09 03:56:07.618960 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618342 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-etc-openvswitch\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.618960 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618369 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-cni-netd\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.618960 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618394 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2217f5b-d7af-42e3-b058-b2e3570e8156-multus-daemon-config\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.618960 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618420 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/747cde48-17af-4cda-ab13-0a621520741c-system-cni-dir\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.618960 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618446 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/747cde48-17af-4cda-ab13-0a621520741c-cnibin\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.618960 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618475 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/747cde48-17af-4cda-ab13-0a621520741c-os-release\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.618960 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618505 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07737c3c-20ee-414b-8dde-b9661df9e485-iptables-alerter-script\") pod \"iptables-alerter-kc98t\" (UID: \"07737c3c-20ee-414b-8dde-b9661df9e485\") " pod="openshift-network-operator/iptables-alerter-kc98t" Jul 09 03:56:07.618960 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618530 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-run-systemd\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.618960 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618555 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-log-socket\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.618960 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618581 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3cd86c1a-eab6-4284-9182-8c034842eaf7-ovnkube-config\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.618960 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618607 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-etc-selinux\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.619660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618632 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-var-lib-kubelet\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.619660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618658 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl5ct\" (UniqueName: \"kubernetes.io/projected/a2217f5b-d7af-42e3-b058-b2e3570e8156-kube-api-access-zl5ct\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.619660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618684 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-lib-modules\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.619660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618711 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-multus-socket-dir-parent\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.619660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618742 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-sysctl-conf\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.619660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618766 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-sys\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.619660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618791 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-tuned\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.619660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618816 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrjxg\" (UniqueName: \"kubernetes.io/projected/e2445180-4ff5-44bb-8e74-1b581138eeca-kube-api-access-qrjxg\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.619660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618841 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23a13f6-edd0-42f2-a03e-ba0b1cf7186d-tmp-dir\") pod \"node-resolver-ltzfl\" (UID: \"e23a13f6-edd0-42f2-a03e-ba0b1cf7186d\") " pod="openshift-dns/node-resolver-ltzfl" Jul 09 03:56:07.619660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618890 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/388dfe09-6614-4173-a31b-65185f254dc4-konnectivity-ca\") pod \"konnectivity-agent-vf56w\" (UID: \"388dfe09-6614-4173-a31b-65185f254dc4\") " pod="kube-system/konnectivity-agent-vf56w" Jul 09 03:56:07.619660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618917 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pc86\" (UniqueName: \"kubernetes.io/projected/747cde48-17af-4cda-ab13-0a621520741c-kube-api-access-9pc86\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.619660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618966 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms2gs\" (UniqueName: \"kubernetes.io/projected/3cd86c1a-eab6-4284-9182-8c034842eaf7-kube-api-access-ms2gs\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.619660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.618994 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-run-openvswitch\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.619660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619021 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-node-log\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.619660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619047 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-run-netns\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.620447 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619072 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-sysconfig\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.620447 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619096 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-sysctl-d\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.620447 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619121 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvhz4\" (UniqueName: \"kubernetes.io/projected/15fb2067-4603-4c89-95d2-92f7f719166a-kube-api-access-zvhz4\") pod \"node-ca-gn78p\" (UID: \"15fb2067-4603-4c89-95d2-92f7f719166a\") " pod="openshift-image-registry/node-ca-gn78p" Jul 09 03:56:07.620447 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619150 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-run-ovn-kubernetes\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.620447 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619178 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret\") pod \"global-pull-secret-syncer-28sj8\" (UID: \"0d4983fe-8ee1-40b3-89f6-3508333126b3\") " pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:07.620447 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619205 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-sys-fs\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.620447 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619230 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-etc-kubernetes\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.620447 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619255 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-host\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.620447 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619280 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-cni-bin\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.620447 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619308 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-var-lib-cni-multus\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.620447 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619332 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-run\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.620447 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619359 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdsm6\" (UniqueName: \"kubernetes.io/projected/e23a13f6-edd0-42f2-a03e-ba0b1cf7186d-kube-api-access-tdsm6\") pod \"node-resolver-ltzfl\" (UID: \"e23a13f6-edd0-42f2-a03e-ba0b1cf7186d\") " pod="openshift-dns/node-resolver-ltzfl" Jul 09 03:56:07.620447 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619411 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/747cde48-17af-4cda-ab13-0a621520741c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.620447 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619452 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/747cde48-17af-4cda-ab13-0a621520741c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.621171 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619474 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/747cde48-17af-4cda-ab13-0a621520741c-cnibin\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.621171 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619484 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-multus-conf-dir\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.621171 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619523 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr2f5\" (UniqueName: \"kubernetes.io/projected/579aea83-7020-4eb2-9cc9-eb6318890a3e-kube-api-access-hr2f5\") pod \"network-metrics-daemon-fvb66\" (UID: \"579aea83-7020-4eb2-9cc9-eb6318890a3e\") " pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:07.621171 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619528 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-multus-conf-dir\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.621171 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619556 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07737c3c-20ee-414b-8dde-b9661df9e485-host-slash\") pod \"iptables-alerter-kc98t\" (UID: \"07737c3c-20ee-414b-8dde-b9661df9e485\") " pod="openshift-network-operator/iptables-alerter-kc98t" Jul 09 03:56:07.621171 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619584 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-slash\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.621171 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619614 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.621171 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619701 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-system-cni-dir\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.621171 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619732 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-device-dir\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.621171 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619756 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/747cde48-17af-4cda-ab13-0a621520741c-cni-binary-copy\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.621171 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619784 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/747cde48-17af-4cda-ab13-0a621520741c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.621171 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619812 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-run-k8s-cni-cncf-io\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.621171 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619840 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-hostroot\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.621171 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619868 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-run-multus-certs\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.621913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619922 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-kubernetes\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.621913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.620060 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-kubernetes\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.621913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.620122 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-registration-dir\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.621913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.620189 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-os-release\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.621913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.620248 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07737c3c-20ee-414b-8dde-b9661df9e485-iptables-alerter-script\") pod \"iptables-alerter-kc98t\" (UID: \"07737c3c-20ee-414b-8dde-b9661df9e485\") " pod="openshift-network-operator/iptables-alerter-kc98t" Jul 09 03:56:07.621913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.620313 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-modprobe-d\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.621913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.620333 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-run-systemd\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.621913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.620361 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-var-lib-openvswitch\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.621913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.620378 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-log-socket\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.621913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.620412 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-systemd\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.621913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.620458 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-systemd-units\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.621913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.620500 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-run-ovn\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.621913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.620693 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d4983fe-8ee1-40b3-89f6-3508333126b3-dbus\") pod \"global-pull-secret-syncer-28sj8\" (UID: \"0d4983fe-8ee1-40b3-89f6-3508333126b3\") " pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:07.621913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.620763 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.621913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.620869 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-socket-dir\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.622677 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.620916 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-var-lib-cni-bin\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.622677 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.620916 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3cd86c1a-eab6-4284-9182-8c034842eaf7-ovnkube-config\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.622677 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.621047 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-etc-selinux\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.622677 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.621095 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-var-lib-kubelet\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.622677 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.621223 2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 03:56:07.622677 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.621424 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-lib-modules\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.622677 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.621488 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-multus-socket-dir-parent\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.622677 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.621621 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-sysctl-conf\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.622677 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.621680 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-sys\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.624052 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.623750 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23a13f6-edd0-42f2-a03e-ba0b1cf7186d-tmp-dir\") pod \"node-resolver-ltzfl\" (UID: \"e23a13f6-edd0-42f2-a03e-ba0b1cf7186d\") " pod="openshift-dns/node-resolver-ltzfl" Jul 09 03:56:07.624439 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.624417 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/388dfe09-6614-4173-a31b-65185f254dc4-konnectivity-ca\") pod \"konnectivity-agent-vf56w\" (UID: \"388dfe09-6614-4173-a31b-65185f254dc4\") " pod="kube-system/konnectivity-agent-vf56w" Jul 09 03:56:07.624990 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.624944 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-run-openvswitch\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.625105 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.625013 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-node-log\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.625105 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.625070 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-run-netns\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.625279 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.625129 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-sysconfig\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.625279 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.625235 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-sysctl-d\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.625468 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:07.625447 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 03:56:07.625537 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:07.625480 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 03:56:07.625537 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:07.625496 2571 projected.go:196] Error preparing data for projected volume kube-api-access-xmrj4 for pod openshift-network-diagnostics/network-check-target-c7ws6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 03:56:07.625643 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:07.625583 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4 podName:087e5e53-db4d-4eed-bad5-fb6949ff07c3 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:08.125537697 +0000 UTC m=+2.984408778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xmrj4" (UniqueName: "kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4") pod "network-check-target-c7ws6" (UID: "087e5e53-db4d-4eed-bad5-fb6949ff07c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 03:56:07.625716 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.625649 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2445180-4ff5-44bb-8e74-1b581138eeca-tmp\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.626809 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:07.625771 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 03:56:07.626809 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.625811 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-run-ovn-kubernetes\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.626809 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:07.625832 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs podName:579aea83-7020-4eb2-9cc9-eb6318890a3e nodeName:}" failed. No retries permitted until 2026-07-09 03:56:08.125813312 +0000 UTC m=+2.984684393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs") pod "network-metrics-daemon-fvb66" (UID: "579aea83-7020-4eb2-9cc9-eb6318890a3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 03:56:07.626809 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.625894 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-sys-fs\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.626809 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.625958 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-kubelet\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.626809 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:07.626028 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 03:56:07.626809 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:07.626076 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret podName:0d4983fe-8ee1-40b3-89f6-3508333126b3 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:08.126059212 +0000 UTC m=+2.984930298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret") pod "global-pull-secret-syncer-28sj8" (UID: "0d4983fe-8ee1-40b3-89f6-3508333126b3") : object "kube-system"/"original-pull-secret" not registered Jul 09 03:56:07.626809 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.626117 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-etc-kubernetes\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.626809 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.626171 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-host\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.626809 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.626258 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3cd86c1a-eab6-4284-9182-8c034842eaf7-env-overrides\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.626809 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.626461 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-var-lib-cni-multus\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.626809 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.626517 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-run\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.628049 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.627544 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2217f5b-d7af-42e3-b058-b2e3570e8156-multus-daemon-config\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.628109 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.628058 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3cd86c1a-eab6-4284-9182-8c034842eaf7-ovnkube-script-lib\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.628399 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.628375 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-cnibin\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.630831 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.628829 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2217f5b-d7af-42e3-b058-b2e3570e8156-cni-binary-copy\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.630831 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.628908 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d4983fe-8ee1-40b3-89f6-3508333126b3-kubelet-config\") pod \"global-pull-secret-syncer-28sj8\" (UID: \"0d4983fe-8ee1-40b3-89f6-3508333126b3\") " pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:07.630831 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.629182 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-run-netns\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.630831 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.629274 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-multus-cni-dir\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.630831 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.629331 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2445180-4ff5-44bb-8e74-1b581138eeca-var-lib-kubelet\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.630831 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.629376 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15fb2067-4603-4c89-95d2-92f7f719166a-host\") pod \"node-ca-gn78p\" (UID: \"15fb2067-4603-4c89-95d2-92f7f719166a\") " pod="openshift-image-registry/node-ca-gn78p" Jul 09 03:56:07.630831 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.629727 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e2445180-4ff5-44bb-8e74-1b581138eeca-etc-tuned\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.630831 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.629830 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/15fb2067-4603-4c89-95d2-92f7f719166a-serviceca\") pod \"node-ca-gn78p\" (UID: \"15fb2067-4603-4c89-95d2-92f7f719166a\") " pod="openshift-image-registry/node-ca-gn78p" Jul 09 03:56:07.630831 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.630705 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/650443f7-bb0e-4c54-b0bb-996c52116d90-device-dir\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.630831 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.630738 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/747cde48-17af-4cda-ab13-0a621520741c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.630831 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.630823 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/747cde48-17af-4cda-ab13-0a621520741c-system-cni-dir\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.631486 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.631076 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-etc-openvswitch\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.631486 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.631149 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.631486 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.631196 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07737c3c-20ee-414b-8dde-b9661df9e485-host-slash\") pod \"iptables-alerter-kc98t\" (UID: \"07737c3c-20ee-414b-8dde-b9661df9e485\") " pod="openshift-network-operator/iptables-alerter-kc98t" Jul 09 03:56:07.631486 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.631240 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-slash\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.631486 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.631292 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/747cde48-17af-4cda-ab13-0a621520741c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.631749 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.619619 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/747cde48-17af-4cda-ab13-0a621520741c-os-release\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.631749 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.631632 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-system-cni-dir\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.631749 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.631688 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-cni-netd\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.631749 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.631736 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-hostroot\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.632379 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.632353 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/747cde48-17af-4cda-ab13-0a621520741c-cni-binary-copy\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.632543 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.632522 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/747cde48-17af-4cda-ab13-0a621520741c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.632612 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.632586 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-run-k8s-cni-cncf-io\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.637849 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.637816 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl5ct\" (UniqueName: \"kubernetes.io/projected/a2217f5b-d7af-42e3-b058-b2e3570e8156-kube-api-access-zl5ct\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.638425 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.638336 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/388dfe09-6614-4173-a31b-65185f254dc4-agent-certs\") pod \"konnectivity-agent-vf56w\" (UID: \"388dfe09-6614-4173-a31b-65185f254dc4\") " pod="kube-system/konnectivity-agent-vf56w" Jul 09 03:56:07.640631 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.639413 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2217f5b-d7af-42e3-b058-b2e3570e8156-host-run-multus-certs\") pod \"multus-59wnp\" (UID: \"a2217f5b-d7af-42e3-b058-b2e3570e8156\") " pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.640631 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.625773 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3cd86c1a-eab6-4284-9182-8c034842eaf7-host-cni-bin\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.640631 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.639583 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms2gs\" (UniqueName: \"kubernetes.io/projected/3cd86c1a-eab6-4284-9182-8c034842eaf7-kube-api-access-ms2gs\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.640631 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.640574 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdsm6\" (UniqueName: \"kubernetes.io/projected/e23a13f6-edd0-42f2-a03e-ba0b1cf7186d-kube-api-access-tdsm6\") pod \"node-resolver-ltzfl\" (UID: \"e23a13f6-edd0-42f2-a03e-ba0b1cf7186d\") " pod="openshift-dns/node-resolver-ltzfl" Jul 09 03:56:07.640631 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.640608 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvhz4\" (UniqueName: \"kubernetes.io/projected/15fb2067-4603-4c89-95d2-92f7f719166a-kube-api-access-zvhz4\") pod \"node-ca-gn78p\" (UID: \"15fb2067-4603-4c89-95d2-92f7f719166a\") " pod="openshift-image-registry/node-ca-gn78p" Jul 09 03:56:07.643894 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.642925 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrjxg\" (UniqueName: \"kubernetes.io/projected/e2445180-4ff5-44bb-8e74-1b581138eeca-kube-api-access-qrjxg\") pod \"tuned-62cz2\" (UID: \"e2445180-4ff5-44bb-8e74-1b581138eeca\") " pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.643894 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.643274 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3cd86c1a-eab6-4284-9182-8c034842eaf7-ovn-node-metrics-cert\") pod \"ovnkube-node-mwnbk\" (UID: \"3cd86c1a-eab6-4284-9182-8c034842eaf7\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.644876 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.644557 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54h2c\" (UniqueName: \"kubernetes.io/projected/07737c3c-20ee-414b-8dde-b9661df9e485-kube-api-access-54h2c\") pod \"iptables-alerter-kc98t\" (UID: \"07737c3c-20ee-414b-8dde-b9661df9e485\") " pod="openshift-network-operator/iptables-alerter-kc98t" Jul 09 03:56:07.644876 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.644768 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt7w6\" (UniqueName: \"kubernetes.io/projected/650443f7-bb0e-4c54-b0bb-996c52116d90-kube-api-access-tt7w6\") pod \"aws-ebs-csi-driver-node-5rk44\" (UID: \"650443f7-bb0e-4c54-b0bb-996c52116d90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.645453 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.645333 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" event={"ID":"3abb2fe191c6b50ca1da05d67be83670","Type":"ContainerStarted","Data":"009af49f81a398d5261f950f118a77616f145583a08f7e1b582311bcec62f07a"} Jul 09 03:56:07.645563 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.645538 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pc86\" (UniqueName: \"kubernetes.io/projected/747cde48-17af-4cda-ab13-0a621520741c-kube-api-access-9pc86\") pod \"multus-additional-cni-plugins-bb7z2\" (UID: \"747cde48-17af-4cda-ab13-0a621520741c\") " pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.646875 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.646460 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr2f5\" (UniqueName: \"kubernetes.io/projected/579aea83-7020-4eb2-9cc9-eb6318890a3e-kube-api-access-hr2f5\") pod \"network-metrics-daemon-fvb66\" (UID: \"579aea83-7020-4eb2-9cc9-eb6318890a3e\") " pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:07.647708 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.647680 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" event={"ID":"77031bd827de1cac5285b54b00e4a311","Type":"ContainerStarted","Data":"fb514f5e52a33ec9edf5daec06acba2c69a4e5d00740b252edb487861f668b85"} Jul 09 03:56:07.795191 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.795108 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-59wnp" Jul 09 03:56:07.805723 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:07.805689 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2217f5b_d7af_42e3_b058_b2e3570e8156.slice/crio-bab5aa1e671d737a8d626a203bcc6c3c490f7a46f8e0d580325b0e9211d73d44 WatchSource:0}: Error finding container bab5aa1e671d737a8d626a203bcc6c3c490f7a46f8e0d580325b0e9211d73d44: Status 404 returned error can't find the container with id bab5aa1e671d737a8d626a203bcc6c3c490f7a46f8e0d580325b0e9211d73d44 Jul 09 03:56:07.806402 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.806352 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kc98t" Jul 09 03:56:07.816484 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:07.816456 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07737c3c_20ee_414b_8dde_b9661df9e485.slice/crio-f8dc1d57d1465f12b2a42dd2dce9589a8d6df613e4a36954844d7f58f1a12704 WatchSource:0}: Error finding container f8dc1d57d1465f12b2a42dd2dce9589a8d6df613e4a36954844d7f58f1a12704: Status 404 returned error can't find the container with id f8dc1d57d1465f12b2a42dd2dce9589a8d6df613e4a36954844d7f58f1a12704 Jul 09 03:56:07.819123 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.819102 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-62cz2" Jul 09 03:56:07.823358 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.823205 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ltzfl" Jul 09 03:56:07.830493 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:07.830465 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2445180_4ff5_44bb_8e74_1b581138eeca.slice/crio-fe2597e5ebdcb927c68c93230498ff931013d8449cdcfa1390da2d753472b206 WatchSource:0}: Error finding container fe2597e5ebdcb927c68c93230498ff931013d8449cdcfa1390da2d753472b206: Status 404 returned error can't find the container with id fe2597e5ebdcb927c68c93230498ff931013d8449cdcfa1390da2d753472b206 Jul 09 03:56:07.831707 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.831656 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gn78p" Jul 09 03:56:07.841337 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:07.841301 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fb2067_4603_4c89_95d2_92f7f719166a.slice/crio-e15147b505c326f592a5dd299521597e05a3dbea7783e7befb41cd9de0b9696b WatchSource:0}: Error finding container e15147b505c326f592a5dd299521597e05a3dbea7783e7befb41cd9de0b9696b: Status 404 returned error can't find the container with id e15147b505c326f592a5dd299521597e05a3dbea7783e7befb41cd9de0b9696b Jul 09 03:56:07.841337 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.841321 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:07.849978 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.849957 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vf56w" Jul 09 03:56:07.850574 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:07.850550 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd86c1a_eab6_4284_9182_8c034842eaf7.slice/crio-e85492ebf41e974f893f12c37c5f0f7a43684f4011f62b6b06879aa09175b221 WatchSource:0}: Error finding container e85492ebf41e974f893f12c37c5f0f7a43684f4011f62b6b06879aa09175b221: Status 404 returned error can't find the container with id e85492ebf41e974f893f12c37c5f0f7a43684f4011f62b6b06879aa09175b221 Jul 09 03:56:07.858913 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:07.858889 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388dfe09_6614_4173_a31b_65185f254dc4.slice/crio-a51ff0e6e6bfd9adc0bdf4c58522d8fcea3067fe5a7e049e4beaf2d5c6720847 WatchSource:0}: Error finding container a51ff0e6e6bfd9adc0bdf4c58522d8fcea3067fe5a7e049e4beaf2d5c6720847: Status 404 returned error can't find the container with id a51ff0e6e6bfd9adc0bdf4c58522d8fcea3067fe5a7e049e4beaf2d5c6720847 Jul 09 03:56:07.859832 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.859675 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" Jul 09 03:56:07.867340 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:07.867190 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bb7z2" Jul 09 03:56:07.870823 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:07.870800 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod650443f7_bb0e_4c54_b0bb_996c52116d90.slice/crio-f0b39e58b8b3b7aafa45cf95698a420a2a0fe0a74ec87bc8a5a17b28c7e91944 WatchSource:0}: Error finding container f0b39e58b8b3b7aafa45cf95698a420a2a0fe0a74ec87bc8a5a17b28c7e91944: Status 404 returned error can't find the container with id f0b39e58b8b3b7aafa45cf95698a420a2a0fe0a74ec87bc8a5a17b28c7e91944 Jul 09 03:56:07.876683 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:07.876658 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747cde48_17af_4cda_ab13_0a621520741c.slice/crio-a69fbd813dd488dc820ae88a1ae0e618c97ffa53559fef76eaa4439dcdbaa325 WatchSource:0}: Error finding container a69fbd813dd488dc820ae88a1ae0e618c97ffa53559fef76eaa4439dcdbaa325: Status 404 returned error can't find the container with id a69fbd813dd488dc820ae88a1ae0e618c97ffa53559fef76eaa4439dcdbaa325 Jul 09 03:56:08.224245 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:08.224189 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret\") pod \"global-pull-secret-syncer-28sj8\" (UID: \"0d4983fe-8ee1-40b3-89f6-3508333126b3\") " pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:08.224466 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:08.224258 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrj4\" (UniqueName: \"kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4\") pod \"network-check-target-c7ws6\" (UID: \"087e5e53-db4d-4eed-bad5-fb6949ff07c3\") " pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:08.224466 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:08.224289 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs\") pod \"network-metrics-daemon-fvb66\" (UID: \"579aea83-7020-4eb2-9cc9-eb6318890a3e\") " pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:08.224466 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:08.224353 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 03:56:08.224466 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:08.224415 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 03:56:08.224466 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:08.224439 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 03:56:08.224466 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:08.224465 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 03:56:08.224715 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:08.224482 2571 projected.go:196] Error preparing data for projected volume kube-api-access-xmrj4 for pod openshift-network-diagnostics/network-check-target-c7ws6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 03:56:08.224715 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:08.224444 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret podName:0d4983fe-8ee1-40b3-89f6-3508333126b3 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:09.224422405 +0000 UTC m=+4.083293491 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret") pod "global-pull-secret-syncer-28sj8" (UID: "0d4983fe-8ee1-40b3-89f6-3508333126b3") : object "kube-system"/"original-pull-secret" not registered Jul 09 03:56:08.224715 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:08.224536 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs podName:579aea83-7020-4eb2-9cc9-eb6318890a3e nodeName:}" failed. No retries permitted until 2026-07-09 03:56:09.224521841 +0000 UTC m=+4.083392921 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs") pod "network-metrics-daemon-fvb66" (UID: "579aea83-7020-4eb2-9cc9-eb6318890a3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 03:56:08.224715 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:08.224548 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4 podName:087e5e53-db4d-4eed-bad5-fb6949ff07c3 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:09.224541846 +0000 UTC m=+4.083412926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xmrj4" (UniqueName: "kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4") pod "network-check-target-c7ws6" (UID: "087e5e53-db4d-4eed-bad5-fb6949ff07c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 03:56:08.554950 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:08.554833 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 03:51:08 +0000 UTC" deadline="2028-01-20 04:35:56.858190616 +0000 UTC" Jul 09 03:56:08.554950 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:08.554886 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13440h39m48.303309808s" Jul 09 03:56:08.651710 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:08.651668 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kc98t" event={"ID":"07737c3c-20ee-414b-8dde-b9661df9e485","Type":"ContainerStarted","Data":"f8dc1d57d1465f12b2a42dd2dce9589a8d6df613e4a36954844d7f58f1a12704"} Jul 09 03:56:08.653707 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:08.653679 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bb7z2" event={"ID":"747cde48-17af-4cda-ab13-0a621520741c","Type":"ContainerStarted","Data":"a69fbd813dd488dc820ae88a1ae0e618c97ffa53559fef76eaa4439dcdbaa325"} Jul 09 03:56:08.655804 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:08.655751 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vf56w" event={"ID":"388dfe09-6614-4173-a31b-65185f254dc4","Type":"ContainerStarted","Data":"a51ff0e6e6bfd9adc0bdf4c58522d8fcea3067fe5a7e049e4beaf2d5c6720847"} Jul 09 03:56:08.657311 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:08.657220 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" event={"ID":"3cd86c1a-eab6-4284-9182-8c034842eaf7","Type":"ContainerStarted","Data":"e85492ebf41e974f893f12c37c5f0f7a43684f4011f62b6b06879aa09175b221"} Jul 09 03:56:08.658698 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:08.658637 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gn78p" event={"ID":"15fb2067-4603-4c89-95d2-92f7f719166a","Type":"ContainerStarted","Data":"e15147b505c326f592a5dd299521597e05a3dbea7783e7befb41cd9de0b9696b"} Jul 09 03:56:08.660234 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:08.660207 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ltzfl" event={"ID":"e23a13f6-edd0-42f2-a03e-ba0b1cf7186d","Type":"ContainerStarted","Data":"190eff07872dabaa96211b32707c393836f6d535ebf20cd52248ede84696d68d"} Jul 09 03:56:08.661646 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:08.661623 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-59wnp" event={"ID":"a2217f5b-d7af-42e3-b058-b2e3570e8156","Type":"ContainerStarted","Data":"bab5aa1e671d737a8d626a203bcc6c3c490f7a46f8e0d580325b0e9211d73d44"} Jul 09 03:56:08.663128 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:08.663096 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" event={"ID":"650443f7-bb0e-4c54-b0bb-996c52116d90","Type":"ContainerStarted","Data":"f0b39e58b8b3b7aafa45cf95698a420a2a0fe0a74ec87bc8a5a17b28c7e91944"} Jul 09 03:56:08.664825 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:08.664799 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-62cz2" event={"ID":"e2445180-4ff5-44bb-8e74-1b581138eeca","Type":"ContainerStarted","Data":"fe2597e5ebdcb927c68c93230498ff931013d8449cdcfa1390da2d753472b206"} Jul 09 03:56:09.235291 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:09.234992 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrj4\" (UniqueName: \"kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4\") pod \"network-check-target-c7ws6\" (UID: \"087e5e53-db4d-4eed-bad5-fb6949ff07c3\") " pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:09.235291 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:09.235052 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs\") pod \"network-metrics-daemon-fvb66\" (UID: \"579aea83-7020-4eb2-9cc9-eb6318890a3e\") " pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:09.235291 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:09.235113 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret\") pod \"global-pull-secret-syncer-28sj8\" (UID: \"0d4983fe-8ee1-40b3-89f6-3508333126b3\") " pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:09.235291 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:09.235216 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 03:56:09.235291 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:09.235253 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 03:56:09.235291 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:09.235256 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 03:56:09.235291 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:09.235226 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 03:56:09.235759 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:09.235268 2571 projected.go:196] Error preparing data for projected volume kube-api-access-xmrj4 for pod openshift-network-diagnostics/network-check-target-c7ws6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 03:56:09.235759 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:09.235320 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs podName:579aea83-7020-4eb2-9cc9-eb6318890a3e nodeName:}" failed. No retries permitted until 2026-07-09 03:56:11.235304937 +0000 UTC m=+6.094176016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs") pod "network-metrics-daemon-fvb66" (UID: "579aea83-7020-4eb2-9cc9-eb6318890a3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 03:56:09.235759 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:09.235335 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret podName:0d4983fe-8ee1-40b3-89f6-3508333126b3 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:11.235328549 +0000 UTC m=+6.094199629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret") pod "global-pull-secret-syncer-28sj8" (UID: "0d4983fe-8ee1-40b3-89f6-3508333126b3") : object "kube-system"/"original-pull-secret" not registered Jul 09 03:56:09.235759 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:09.235347 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4 podName:087e5e53-db4d-4eed-bad5-fb6949ff07c3 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:11.235341081 +0000 UTC m=+6.094212160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xmrj4" (UniqueName: "kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4") pod "network-check-target-c7ws6" (UID: "087e5e53-db4d-4eed-bad5-fb6949ff07c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 03:56:09.555583 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:09.555516 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 03:51:08 +0000 UTC" deadline="2027-12-19 00:47:10.02367322 +0000 UTC" Jul 09 03:56:09.555583 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:09.555556 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12668h51m0.46812235s" Jul 09 03:56:09.629590 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:09.626232 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:09.629590 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:09.626486 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:09.629590 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:09.626667 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:09.629590 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:09.626747 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:09.629590 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:09.626832 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:09.629590 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:09.627056 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:09.688433 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:09.688393 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" event={"ID":"77031bd827de1cac5285b54b00e4a311","Type":"ContainerStarted","Data":"e4b8b78cf93705c0226df01062a397e125acb12dc6d13299b17556c4a7dd8db0"} Jul 09 03:56:10.695209 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:10.695117 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" event={"ID":"3abb2fe191c6b50ca1da05d67be83670","Type":"ContainerStarted","Data":"c0363e35c5979150e7b17459c9af169208e16272f8847cc43dfb61bb88ec3236"} Jul 09 03:56:10.709901 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:10.709819 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" podStartSLOduration=3.709797648 podStartE2EDuration="3.709797648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:09.70587651 +0000 UTC m=+4.564747615" watchObservedRunningTime="2026-07-09 03:56:10.709797648 +0000 UTC m=+5.568668752" Jul 09 03:56:11.256833 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:11.255816 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret\") pod \"global-pull-secret-syncer-28sj8\" (UID: \"0d4983fe-8ee1-40b3-89f6-3508333126b3\") " pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:11.256833 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:11.255876 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrj4\" (UniqueName: \"kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4\") pod \"network-check-target-c7ws6\" (UID: \"087e5e53-db4d-4eed-bad5-fb6949ff07c3\") " pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:11.256833 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:11.255926 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs\") pod \"network-metrics-daemon-fvb66\" (UID: \"579aea83-7020-4eb2-9cc9-eb6318890a3e\") " pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:11.256833 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:11.256083 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 03:56:11.256833 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:11.256148 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs podName:579aea83-7020-4eb2-9cc9-eb6318890a3e nodeName:}" failed. No retries permitted until 2026-07-09 03:56:15.256128273 +0000 UTC m=+10.114999376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs") pod "network-metrics-daemon-fvb66" (UID: "579aea83-7020-4eb2-9cc9-eb6318890a3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 03:56:11.256833 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:11.256566 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 03:56:11.256833 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:11.256616 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret podName:0d4983fe-8ee1-40b3-89f6-3508333126b3 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:15.256601327 +0000 UTC m=+10.115472424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret") pod "global-pull-secret-syncer-28sj8" (UID: "0d4983fe-8ee1-40b3-89f6-3508333126b3") : object "kube-system"/"original-pull-secret" not registered Jul 09 03:56:11.256833 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:11.256696 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 03:56:11.256833 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:11.256714 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 03:56:11.256833 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:11.256727 2571 projected.go:196] Error preparing data for projected volume kube-api-access-xmrj4 for pod openshift-network-diagnostics/network-check-target-c7ws6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 03:56:11.256833 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:11.256792 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4 podName:087e5e53-db4d-4eed-bad5-fb6949ff07c3 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:15.256779384 +0000 UTC m=+10.115650480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xmrj4" (UniqueName: "kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4") pod "network-check-target-c7ws6" (UID: "087e5e53-db4d-4eed-bad5-fb6949ff07c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 03:56:11.623388 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:11.622579 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:11.623388 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:11.622732 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:11.623388 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:11.623150 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:11.623388 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:11.623254 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:11.624059 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:11.623807 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:11.624059 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:11.623964 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:12.705321 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:12.704714 2571 generic.go:356] "Generic (PLEG): container finished" podID="3abb2fe191c6b50ca1da05d67be83670" containerID="c0363e35c5979150e7b17459c9af169208e16272f8847cc43dfb61bb88ec3236" exitCode=0 Jul 09 03:56:12.705321 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:12.704754 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" event={"ID":"3abb2fe191c6b50ca1da05d67be83670","Type":"ContainerDied","Data":"c0363e35c5979150e7b17459c9af169208e16272f8847cc43dfb61bb88ec3236"} Jul 09 03:56:13.620487 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:13.619945 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:13.620487 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:13.619983 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:13.620487 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:13.619945 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:13.620487 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:13.620090 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:13.620487 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:13.620140 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:13.620487 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:13.620220 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:15.300370 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:15.300324 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrj4\" (UniqueName: \"kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4\") pod \"network-check-target-c7ws6\" (UID: \"087e5e53-db4d-4eed-bad5-fb6949ff07c3\") " pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:15.300784 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:15.300383 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs\") pod \"network-metrics-daemon-fvb66\" (UID: \"579aea83-7020-4eb2-9cc9-eb6318890a3e\") " pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:15.300784 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:15.300436 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret\") pod \"global-pull-secret-syncer-28sj8\" (UID: \"0d4983fe-8ee1-40b3-89f6-3508333126b3\") " pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:15.300784 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:15.300582 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 03:56:15.300784 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:15.300648 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret podName:0d4983fe-8ee1-40b3-89f6-3508333126b3 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:23.300630248 +0000 UTC m=+18.159501329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret") pod "global-pull-secret-syncer-28sj8" (UID: "0d4983fe-8ee1-40b3-89f6-3508333126b3") : object "kube-system"/"original-pull-secret" not registered Jul 09 03:56:15.300784 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:15.300743 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 03:56:15.300784 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:15.300764 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 03:56:15.300784 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:15.300777 2571 projected.go:196] Error preparing data for projected volume kube-api-access-xmrj4 for pod openshift-network-diagnostics/network-check-target-c7ws6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 03:56:15.301172 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:15.300821 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4 podName:087e5e53-db4d-4eed-bad5-fb6949ff07c3 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:23.300808317 +0000 UTC m=+18.159679411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xmrj4" (UniqueName: "kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4") pod "network-check-target-c7ws6" (UID: "087e5e53-db4d-4eed-bad5-fb6949ff07c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 03:56:15.301172 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:15.300897 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 03:56:15.301172 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:15.300928 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs podName:579aea83-7020-4eb2-9cc9-eb6318890a3e nodeName:}" failed. No retries permitted until 2026-07-09 03:56:23.300918114 +0000 UTC m=+18.159789209 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs") pod "network-metrics-daemon-fvb66" (UID: "579aea83-7020-4eb2-9cc9-eb6318890a3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 03:56:15.621454 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:15.621417 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:15.621633 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:15.621527 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:15.621866 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:15.621837 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:15.621970 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:15.621947 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:15.622114 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:15.622088 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:15.622191 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:15.622169 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:17.622242 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:17.621820 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:17.622242 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:17.621980 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:17.622242 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:17.622071 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:17.622242 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:17.622172 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:17.623048 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:17.622924 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:17.623048 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:17.623010 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:19.620989 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.620941 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:19.621477 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.620957 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:19.621477 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:19.621069 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:19.621477 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.621089 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:19.621477 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:19.621208 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:19.621477 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:19.621278 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:19.722004 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.721837 2571 generic.go:356] "Generic (PLEG): container finished" podID="747cde48-17af-4cda-ab13-0a621520741c" containerID="03f3aa95b51e752062b0fc88451668371cd1958985346c953d85a8136f0b4132" exitCode=0 Jul 09 03:56:19.722004 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.721950 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bb7z2" event={"ID":"747cde48-17af-4cda-ab13-0a621520741c","Type":"ContainerDied","Data":"03f3aa95b51e752062b0fc88451668371cd1958985346c953d85a8136f0b4132"} Jul 09 03:56:19.723991 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.723964 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vf56w" event={"ID":"388dfe09-6614-4173-a31b-65185f254dc4","Type":"ContainerStarted","Data":"f821ef96717ed4b5910103d5dda4a2cc07ede8217b16855a25d59cb701e0df77"} Jul 09 03:56:19.728789 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.728532 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gn78p" event={"ID":"15fb2067-4603-4c89-95d2-92f7f719166a","Type":"ContainerStarted","Data":"6832dfc5e2f9c80c9e235a56a030ed0e707825f3a4fd190ebae46f35756cd5a1"} Jul 09 03:56:19.731166 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.731139 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ltzfl" event={"ID":"e23a13f6-edd0-42f2-a03e-ba0b1cf7186d","Type":"ContainerStarted","Data":"4b2c7a0493d47f45a8aea7f3e4551bfb157065ed219a30d79d9f801cab335776"} Jul 09 03:56:19.733393 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.733361 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" event={"ID":"3abb2fe191c6b50ca1da05d67be83670","Type":"ContainerStarted","Data":"ea54c056a7644346612c8e62a0c74be51f1765b0a4ed8eb5587f9b457bff80e4"} Jul 09 03:56:19.735424 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.735392 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" event={"ID":"650443f7-bb0e-4c54-b0bb-996c52116d90","Type":"ContainerStarted","Data":"7e841528b9a6f8e2b55845e8fee87573a48fd96b38dd8d104cd059c3450921ee"} Jul 09 03:56:19.736999 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.736977 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-62cz2" event={"ID":"e2445180-4ff5-44bb-8e74-1b581138eeca","Type":"ContainerStarted","Data":"b21a06ca51e64afea00d81f79e9523bed569749befd17eb21a3d4f7dfe6461e8"} Jul 09 03:56:19.760365 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.760310 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" podStartSLOduration=12.760292977 podStartE2EDuration="12.760292977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:19.760040276 +0000 UTC m=+14.618911375" watchObservedRunningTime="2026-07-09 03:56:19.760292977 +0000 UTC m=+14.619164081" Jul 09 03:56:19.769174 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.769144 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vf56w" Jul 09 03:56:19.769654 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.769639 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vf56w" Jul 09 03:56:19.792912 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.792847 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-62cz2" podStartSLOduration=2.467555087 podStartE2EDuration="13.79283454s" totalImagesPullingTime="11.325279453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:19.777961735 +0000 UTC m=+14.636832837" watchObservedRunningTime="2026-07-09 03:56:19.79283454 +0000 UTC m=+14.651705640" Jul 09 03:56:19.792998 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.792953 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ltzfl" podStartSLOduration=2.79661079 podStartE2EDuration="13.792949222s" totalImagesPullingTime="10.996338432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:19.792640259 +0000 UTC m=+14.651511360" watchObservedRunningTime="2026-07-09 03:56:19.792949222 +0000 UTC m=+14.651820323" Jul 09 03:56:19.833345 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.833174 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gn78p" podStartSLOduration=2.813179064 podStartE2EDuration="13.833156459s" totalImagesPullingTime="11.019977395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:19.816669997 +0000 UTC m=+14.675541102" watchObservedRunningTime="2026-07-09 03:56:19.833156459 +0000 UTC m=+14.692027560" Jul 09 03:56:19.833529 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:19.833488 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vf56w" podStartSLOduration=2.832667876 podStartE2EDuration="13.833477962s" totalImagesPullingTime="11.000810086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:19.832904847 +0000 UTC m=+14.691775947" watchObservedRunningTime="2026-07-09 03:56:19.833477962 +0000 UTC m=+14.692349063" Jul 09 03:56:20.741063 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:20.741023 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kc98t" event={"ID":"07737c3c-20ee-414b-8dde-b9661df9e485","Type":"ContainerStarted","Data":"1dff6936a9fe3c86a1b7146144432fa5627d6bdcef3cd57cec635d10903ad72f"} Jul 09 03:56:20.755691 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:20.755648 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kc98t" podStartSLOduration=3.740955547 podStartE2EDuration="14.755629974s" totalImagesPullingTime="11.014674427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:20.75499878 +0000 UTC m=+15.613869883" watchObservedRunningTime="2026-07-09 03:56:20.755629974 +0000 UTC m=+15.614501075" Jul 09 03:56:21.620033 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:21.619997 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:21.620191 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:21.619998 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:21.620191 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:21.620133 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:21.620305 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:21.620221 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:21.620305 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:21.620013 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:21.620385 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:21.620310 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:21.743434 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:21.743403 2571 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 03:56:23.364244 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:23.364027 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs\") pod \"network-metrics-daemon-fvb66\" (UID: \"579aea83-7020-4eb2-9cc9-eb6318890a3e\") " pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:23.364733 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:23.364275 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret\") pod \"global-pull-secret-syncer-28sj8\" (UID: \"0d4983fe-8ee1-40b3-89f6-3508333126b3\") " pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:23.364733 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:23.364189 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 03:56:23.364733 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:23.364399 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs podName:579aea83-7020-4eb2-9cc9-eb6318890a3e nodeName:}" failed. No retries permitted until 2026-07-09 03:56:39.364366792 +0000 UTC m=+34.223237885 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs") pod "network-metrics-daemon-fvb66" (UID: "579aea83-7020-4eb2-9cc9-eb6318890a3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 03:56:23.364733 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:23.364313 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrj4\" (UniqueName: \"kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4\") pod \"network-check-target-c7ws6\" (UID: \"087e5e53-db4d-4eed-bad5-fb6949ff07c3\") " pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:23.364733 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:23.364463 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 03:56:23.364733 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:23.364499 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 03:56:23.364733 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:23.364516 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 03:56:23.364733 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:23.364527 2571 projected.go:196] Error preparing data for projected volume kube-api-access-xmrj4 for pod openshift-network-diagnostics/network-check-target-c7ws6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 03:56:23.364733 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:23.364527 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret podName:0d4983fe-8ee1-40b3-89f6-3508333126b3 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:39.364509318 +0000 UTC m=+34.223380413 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret") pod "global-pull-secret-syncer-28sj8" (UID: "0d4983fe-8ee1-40b3-89f6-3508333126b3") : object "kube-system"/"original-pull-secret" not registered Jul 09 03:56:23.364733 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:23.364565 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4 podName:087e5e53-db4d-4eed-bad5-fb6949ff07c3 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:39.364553523 +0000 UTC m=+34.223424601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xmrj4" (UniqueName: "kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4") pod "network-check-target-c7ws6" (UID: "087e5e53-db4d-4eed-bad5-fb6949ff07c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 03:56:23.621031 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:23.620946 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:23.621031 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:23.620982 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:23.621259 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:23.620946 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:23.621259 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:23.621074 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:23.621259 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:23.621183 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:23.621408 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:23.621271 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:25.620722 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:25.620694 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:25.621391 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:25.620798 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:25.621391 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:25.620881 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:25.621391 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:25.620976 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:25.621391 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:25.621009 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:25.621391 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:25.621038 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:27.620198 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:27.620161 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:27.620730 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:27.620160 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:27.620730 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:27.620294 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:27.620730 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:27.620318 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:27.620730 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:27.620432 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:27.620730 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:27.620506 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:29.620639 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:29.620602 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:29.621130 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:29.620612 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:29.621130 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:29.620616 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:29.621130 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:29.620775 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:29.621130 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:29.620917 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:29.621130 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:29.621001 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:30.272650 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:30.272495 2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 03:56:30.594537 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:30.594406 2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T03:56:30.27264576Z","UUID":"02f9d1f5-5960-4549-aa95-51b45ccc1137","Handler":null,"Name":"","Endpoint":""} Jul 09 03:56:30.595912 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:30.595893 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 03:56:30.596014 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:30.595934 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 03:56:30.765130 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:30.765105 2571 generic.go:356] "Generic (PLEG): container finished" podID="747cde48-17af-4cda-ab13-0a621520741c" containerID="1770acb464f57b010f86364a83a515dcf59ff25927d8dc42e9d732a3c531d982" exitCode=0 Jul 09 03:56:30.765637 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:30.765187 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bb7z2" event={"ID":"747cde48-17af-4cda-ab13-0a621520741c","Type":"ContainerDied","Data":"1770acb464f57b010f86364a83a515dcf59ff25927d8dc42e9d732a3c531d982"} Jul 09 03:56:30.768665 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:30.768342 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" event={"ID":"3cd86c1a-eab6-4284-9182-8c034842eaf7","Type":"ContainerStarted","Data":"f5197390f46aa949e391d1b0cd995565c55e662e6740a71b65fbd280132c35ed"} Jul 09 03:56:30.768665 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:30.768422 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" event={"ID":"3cd86c1a-eab6-4284-9182-8c034842eaf7","Type":"ContainerStarted","Data":"dbcda3209b52031c6f6965b117fd726d6eadec3ec32cad64dbaefda89c5610f7"} Jul 09 03:56:30.768665 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:30.768470 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" event={"ID":"3cd86c1a-eab6-4284-9182-8c034842eaf7","Type":"ContainerStarted","Data":"a45ca9f03099b1a10f6009ff7d8c22596c333552046b05f5a5763f43e046f47c"} Jul 09 03:56:30.768665 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:30.768518 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" event={"ID":"3cd86c1a-eab6-4284-9182-8c034842eaf7","Type":"ContainerStarted","Data":"9df3ed65ccf242981773a994665febaeb7a21028ba8798588a31ab33f0e7d44b"} Jul 09 03:56:30.770208 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:30.770184 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-59wnp" event={"ID":"a2217f5b-d7af-42e3-b058-b2e3570e8156","Type":"ContainerStarted","Data":"f74ba0f03b24e01f209d83ae160b98f7f343602aeb0b432ade7eb2de0758e70f"} Jul 09 03:56:30.772516 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:30.772497 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" event={"ID":"650443f7-bb0e-4c54-b0bb-996c52116d90","Type":"ContainerStarted","Data":"0d34f95b9469fe53c40d12715c2560c43413047115ddac22e0cdb229ee357b95"} Jul 09 03:56:30.800018 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:30.799979 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-59wnp" podStartSLOduration=2.471104612 podStartE2EDuration="24.799966557s" totalImagesPullingTime="22.328861945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:30.799634362 +0000 UTC m=+25.658505463" watchObservedRunningTime="2026-07-09 03:56:30.799966557 +0000 UTC m=+25.658837656" Jul 09 03:56:31.620601 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:31.620362 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:31.620781 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:31.620366 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:31.620781 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:31.620711 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:31.620781 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:31.620366 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:31.620942 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:31.620741 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:31.620942 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:31.620849 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:31.777982 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:31.777949 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" event={"ID":"3cd86c1a-eab6-4284-9182-8c034842eaf7","Type":"ContainerStarted","Data":"bc28821adcf7aea431da66344e0e44cef61c3461c8a07c27f3a8d2c0ba25c068"} Jul 09 03:56:31.778396 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:31.778048 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" event={"ID":"3cd86c1a-eab6-4284-9182-8c034842eaf7","Type":"ContainerStarted","Data":"8b7c56ae005de99b4cdab81ed435c9edbaa5d46b7486241a85ce869f0164b0c6"} Jul 09 03:56:32.781266 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:32.781234 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" event={"ID":"650443f7-bb0e-4c54-b0bb-996c52116d90","Type":"ContainerStarted","Data":"d98efaeb696f58f60f9045b975c6a29e20e4351eb28d3f9be5a6bb353fb5b8a2"} Jul 09 03:56:32.783184 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:32.783161 2571 generic.go:356] "Generic (PLEG): container finished" podID="747cde48-17af-4cda-ab13-0a621520741c" containerID="d3fd54246ae73dee3a80a98a2e0a92a9a1494eb9edd2393a0c6a91900a185d21" exitCode=0 Jul 09 03:56:32.783270 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:32.783215 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bb7z2" event={"ID":"747cde48-17af-4cda-ab13-0a621520741c","Type":"ContainerDied","Data":"d3fd54246ae73dee3a80a98a2e0a92a9a1494eb9edd2393a0c6a91900a185d21"} Jul 09 03:56:32.797198 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:32.797160 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rk44" podStartSLOduration=3.473804832 podStartE2EDuration="26.797148361s" totalImagesPullingTime="23.323343529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:56:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:32.79689957 +0000 UTC m=+27.655770671" watchObservedRunningTime="2026-07-09 03:56:32.797148361 +0000 UTC m=+27.656019460" Jul 09 03:56:33.620891 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:33.620840 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:33.621061 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:33.620840 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:33.621061 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:33.620985 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:33.621061 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:33.620867 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:33.621061 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:33.621048 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:33.621270 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:33.621159 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:33.788574 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:33.788544 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" event={"ID":"3cd86c1a-eab6-4284-9182-8c034842eaf7","Type":"ContainerStarted","Data":"b9b452c468d83d1f7651a40811d32bc33148ce92dc733e341615e2e1ada89b61"} Jul 09 03:56:34.793389 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:34.793358 2571 generic.go:356] "Generic (PLEG): container finished" podID="747cde48-17af-4cda-ab13-0a621520741c" containerID="347ac18880af9d8d020226457ce71d0ff5e2417a7be989c1ed6b11fd3e0095de" exitCode=0 Jul 09 03:56:34.793815 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:34.793419 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bb7z2" event={"ID":"747cde48-17af-4cda-ab13-0a621520741c","Type":"ContainerDied","Data":"347ac18880af9d8d020226457ce71d0ff5e2417a7be989c1ed6b11fd3e0095de"} Jul 09 03:56:35.621259 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:35.621004 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:35.621434 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:35.621115 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:35.621434 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:35.621347 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:35.621434 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:35.621139 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:35.621603 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:35.621441 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:35.621603 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:35.621497 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:35.800304 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:35.800269 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" event={"ID":"3cd86c1a-eab6-4284-9182-8c034842eaf7","Type":"ContainerStarted","Data":"4d1c1e754107bec5e8479b0cc535c3a9d7bbcd8df45df3c4810a5123339d62fb"} Jul 09 03:56:35.801014 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:35.800605 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:35.817494 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:35.817470 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:35.825518 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:35.825474 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" podStartSLOduration=7.541940907 podStartE2EDuration="29.825461852s" totalImagesPullingTime="22.283520945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:35.825005372 +0000 UTC m=+30.683876471" watchObservedRunningTime="2026-07-09 03:56:35.825461852 +0000 UTC m=+30.684332951" Jul 09 03:56:36.802464 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:36.802428 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:36.802931 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:36.802562 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:36.819421 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:36.819398 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:56:37.154671 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:37.154638 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fvb66"] Jul 09 03:56:37.154889 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:37.154782 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:37.154946 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:37.154922 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:37.157119 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:37.157094 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-28sj8"] Jul 09 03:56:37.157244 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:37.157198 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:37.157318 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:37.157291 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:37.166091 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:37.166069 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c7ws6"] Jul 09 03:56:37.166213 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:37.166165 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:37.166271 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:37.166252 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:37.824525 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:37.824494 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vf56w" Jul 09 03:56:37.825005 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:37.824633 2571 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 03:56:37.825189 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:37.825166 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vf56w" Jul 09 03:56:38.620368 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:38.620322 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:38.620368 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:38.620370 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:38.620613 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:38.620371 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:38.620613 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:38.620451 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:38.620613 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:38.620574 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:38.620767 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:38.620622 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:39.378448 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:39.378411 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrj4\" (UniqueName: \"kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4\") pod \"network-check-target-c7ws6\" (UID: \"087e5e53-db4d-4eed-bad5-fb6949ff07c3\") " pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:39.378902 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:39.378468 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs\") pod \"network-metrics-daemon-fvb66\" (UID: \"579aea83-7020-4eb2-9cc9-eb6318890a3e\") " pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:39.378902 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:39.378522 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret\") pod \"global-pull-secret-syncer-28sj8\" (UID: \"0d4983fe-8ee1-40b3-89f6-3508333126b3\") " pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:39.378902 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:39.378583 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 03:56:39.378902 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:39.378603 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 03:56:39.378902 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:39.378613 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 03:56:39.378902 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:39.378653 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 03:56:39.378902 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:39.378616 2571 projected.go:196] Error preparing data for projected volume kube-api-access-xmrj4 for pod openshift-network-diagnostics/network-check-target-c7ws6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 03:56:39.378902 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:39.378662 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret podName:0d4983fe-8ee1-40b3-89f6-3508333126b3 nodeName:}" failed. No retries permitted until 2026-07-09 03:57:11.378649901 +0000 UTC m=+66.237520992 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret") pod "global-pull-secret-syncer-28sj8" (UID: "0d4983fe-8ee1-40b3-89f6-3508333126b3") : object "kube-system"/"original-pull-secret" not registered Jul 09 03:56:39.378902 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:39.378737 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs podName:579aea83-7020-4eb2-9cc9-eb6318890a3e nodeName:}" failed. No retries permitted until 2026-07-09 03:57:11.378719897 +0000 UTC m=+66.237590978 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs") pod "network-metrics-daemon-fvb66" (UID: "579aea83-7020-4eb2-9cc9-eb6318890a3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 03:56:39.378902 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:39.378754 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4 podName:087e5e53-db4d-4eed-bad5-fb6949ff07c3 nodeName:}" failed. No retries permitted until 2026-07-09 03:57:11.378744036 +0000 UTC m=+66.237615120 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xmrj4" (UniqueName: "kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4") pod "network-check-target-c7ws6" (UID: "087e5e53-db4d-4eed-bad5-fb6949ff07c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 03:56:40.620188 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:40.620155 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:40.620663 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:40.620154 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:40.620663 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:40.620264 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:40.620663 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:40.620354 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:40.620663 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:40.620155 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:40.620663 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:40.620445 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:41.817126 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:41.817046 2571 generic.go:356] "Generic (PLEG): container finished" podID="747cde48-17af-4cda-ab13-0a621520741c" containerID="eaceda6fc4c114aedefe26f29f67f321505d6e3d336e3c9848c77c88ca38ce3f" exitCode=0 Jul 09 03:56:41.817126 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:41.817105 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bb7z2" event={"ID":"747cde48-17af-4cda-ab13-0a621520741c","Type":"ContainerDied","Data":"eaceda6fc4c114aedefe26f29f67f321505d6e3d336e3c9848c77c88ca38ce3f"} Jul 09 03:56:42.620543 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.620518 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:42.620663 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.620551 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:42.620663 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.620551 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:42.620663 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:42.620608 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28sj8" podUID="0d4983fe-8ee1-40b3-89f6-3508333126b3" Jul 09 03:56:42.620750 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:42.620699 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7ws6" podUID="087e5e53-db4d-4eed-bad5-fb6949ff07c3" Jul 09 03:56:42.620780 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:42.620763 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvb66" podUID="579aea83-7020-4eb2-9cc9-eb6318890a3e" Jul 09 03:56:42.790979 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.790951 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeReady" Jul 09 03:56:42.794030 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.791439 2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 03:56:42.821870 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.821785 2571 generic.go:356] "Generic (PLEG): container finished" podID="747cde48-17af-4cda-ab13-0a621520741c" containerID="cfc25839e07ee9919904cc37bbb5c43d678f04dbe1dc70e30d0e118a0a505e6f" exitCode=0 Jul 09 03:56:42.821870 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.821827 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bb7z2" event={"ID":"747cde48-17af-4cda-ab13-0a621520741c","Type":"ContainerDied","Data":"cfc25839e07ee9919904cc37bbb5c43d678f04dbe1dc70e30d0e118a0a505e6f"} Jul 09 03:56:42.824549 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.824527 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c95f48485-m6qvx"] Jul 09 03:56:42.858204 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.858167 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-s486r"] Jul 09 03:56:42.858323 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.858285 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:42.860833 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.860727 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 03:56:42.860943 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.860932 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 03:56:42.861049 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.861030 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 03:56:42.861222 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.861203 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k8plw\"" Jul 09 03:56:42.868481 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.868232 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 03:56:42.888692 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.888669 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr"] Jul 09 03:56:42.888806 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.888784 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" Jul 09 03:56:42.893878 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.890967 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 09 03:56:42.893878 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.890998 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 09 03:56:42.893878 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.891435 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-km22d\"" Jul 09 03:56:42.910552 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.910534 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c6pgx"] Jul 09 03:56:42.910687 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.910672 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr" Jul 09 03:56:42.913105 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.913072 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-v6fln\"" Jul 09 03:56:42.913105 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.913077 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 09 03:56:42.913257 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.913132 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 09 03:56:42.913257 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.913082 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 09 03:56:42.913257 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.913183 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 09 03:56:42.928619 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.928602 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-ln9ht"] Jul 09 03:56:42.928718 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.928681 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c6pgx" Jul 09 03:56:42.930786 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.930766 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 09 03:56:42.930874 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.930802 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ftwl2\"" Jul 09 03:56:42.931087 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.931073 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 09 03:56:42.946868 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.946835 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn"] Jul 09 03:56:42.946953 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.946937 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:56:42.948982 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.948965 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 09 03:56:42.949052 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.949036 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 09 03:56:42.949249 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.949234 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 09 03:56:42.949350 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.949335 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-56cmc\"" Jul 09 03:56:42.949806 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.949791 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 09 03:56:42.954494 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.954479 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 09 03:56:42.968988 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.968967 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs"] Jul 09 03:56:42.969108 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.969092 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn" Jul 09 03:56:42.971134 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.971115 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 09 03:56:42.971204 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.971119 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 09 03:56:42.971308 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.971293 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 09 03:56:42.971454 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.971442 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 09 03:56:42.971582 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.971569 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7nnkt\"" Jul 09 03:56:42.993173 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.993152 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh"] Jul 09 03:56:42.993261 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.993248 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:56:42.995414 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.995400 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 09 03:56:42.995489 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.995403 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 03:56:42.995721 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.995710 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 03:56:42.996602 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.996586 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5hpxl\"" Jul 09 03:56:42.997128 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:42.997112 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 09 03:56:43.005521 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.005503 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eda93eaa-8da1-4e3c-b104-11f30ea64f25-ca-trust-extracted\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.005583 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.005533 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-bound-sa-token\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.005583 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.005552 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-7n8mr\" (UID: \"26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr" Jul 09 03:56:43.005654 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.005589 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-certificates\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.005654 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.005606 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hw4m\" (UniqueName: \"kubernetes.io/projected/26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e-kube-api-access-4hw4m\") pod \"service-ca-operator-5d46d9ffd4-7n8mr\" (UID: \"26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr" Jul 09 03:56:43.005654 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.005624 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eda93eaa-8da1-4e3c-b104-11f30ea64f25-trusted-ca\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.005654 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.005645 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e-config\") pod \"service-ca-operator-5d46d9ffd4-7n8mr\" (UID: \"26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr" Jul 09 03:56:43.005768 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.005677 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnhb7\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-kube-api-access-mnhb7\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.005768 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.005716 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eda93eaa-8da1-4e3c-b104-11f30ea64f25-image-registry-private-configuration\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.005768 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.005735 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eda93eaa-8da1-4e3c-b104-11f30ea64f25-installation-pull-secrets\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.005866 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.005805 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-s486r\" (UID: \"fc8d4547-833f-48d7-a30b-5c067c926f00\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" Jul 09 03:56:43.005866 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.005838 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.005931 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.005890 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fc8d4547-833f-48d7-a30b-5c067c926f00-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-s486r\" (UID: \"fc8d4547-833f-48d7-a30b-5c067c926f00\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" Jul 09 03:56:43.005931 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.005915 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vv6q\" (UniqueName: \"kubernetes.io/projected/aa89f5c0-4598-4f98-b76c-0748c8a8920b-kube-api-access-5vv6q\") pod \"volume-data-source-validator-5d456fdbcd-c6pgx\" (UID: \"aa89f5c0-4598-4f98-b76c-0748c8a8920b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c6pgx" Jul 09 03:56:43.017055 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.017038 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-cv9wx"] Jul 09 03:56:43.017153 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.017141 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" Jul 09 03:56:43.019220 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.019204 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 09 03:56:43.019322 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.019232 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 09 03:56:43.019570 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.019555 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cdr7m\"" Jul 09 03:56:43.019649 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.019587 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 09 03:56:43.037120 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.037101 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c95f48485-m6qvx"] Jul 09 03:56:43.037197 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.037131 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7955c548df-lznzp"] Jul 09 03:56:43.037236 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.037195 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.039411 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.039395 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 03:56:43.039505 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.039394 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 09 03:56:43.039505 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.039439 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 03:56:43.039505 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.039495 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9w6dk\"" Jul 09 03:56:43.039655 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.039562 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 09 03:56:43.043892 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.043869 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 09 03:56:43.047655 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.047637 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c6pgx"] Jul 09 03:56:43.047744 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.047661 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr"] Jul 09 03:56:43.047744 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.047673 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs"] Jul 09 03:56:43.047744 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.047687 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-9v87h"] Jul 09 03:56:43.047744 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.047738 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:43.050324 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.050308 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 09 03:56:43.050406 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.050335 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 09 03:56:43.050406 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.050311 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kwjkm\"" Jul 09 03:56:43.050565 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.050550 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 03:56:43.050640 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.050576 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 09 03:56:43.050970 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.050950 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 03:56:43.051251 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.051230 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 09 03:56:43.067950 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.067932 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-ln9ht"] Jul 09 03:56:43.068025 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.067960 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-cv9wx"] Jul 09 03:56:43.068025 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.067977 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-s486r"] Jul 09 03:56:43.068025 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.067991 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bcrtc"] Jul 09 03:56:43.068127 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.068071 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9v87h" Jul 09 03:56:43.070347 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.070323 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tps9l\"" Jul 09 03:56:43.070474 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.070365 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 03:56:43.070727 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.070713 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 03:56:43.091514 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.091494 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j9hq8"] Jul 09 03:56:43.091624 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.091610 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:43.093676 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.093659 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q7njr\"" Jul 09 03:56:43.093676 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.093665 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 03:56:43.093775 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.093662 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 03:56:43.106798 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.106779 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eda93eaa-8da1-4e3c-b104-11f30ea64f25-trusted-ca\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.106913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.106808 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e-config\") pod \"service-ca-operator-5d46d9ffd4-7n8mr\" (UID: \"26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr" Jul 09 03:56:43.106913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.106828 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgwzq\" (UniqueName: \"kubernetes.io/projected/0341b993-2f81-4a86-a32d-8c26da6b80bb-kube-api-access-kgwzq\") pod \"console-operator-68b757fb67-ln9ht\" (UID: \"0341b993-2f81-4a86-a32d-8c26da6b80bb\") " pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:56:43.106913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.106845 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec70533-29e6-48ec-967a-e958f1926405-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsjvn\" (UID: \"3ec70533-29e6-48ec-967a-e958f1926405\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn" Jul 09 03:56:43.106913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.106877 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnhb7\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-kube-api-access-mnhb7\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.106913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.106903 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0341b993-2f81-4a86-a32d-8c26da6b80bb-config\") pod \"console-operator-68b757fb67-ln9ht\" (UID: \"0341b993-2f81-4a86-a32d-8c26da6b80bb\") " pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:56:43.107158 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.106927 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7nvc\" (UniqueName: \"kubernetes.io/projected/b9c353a9-991d-42f9-a0f5-bbb80dd30918-kube-api-access-d7nvc\") pod \"cluster-samples-operator-56fcf86bf-ddrwh\" (UID: \"b9c353a9-991d-42f9-a0f5-bbb80dd30918\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" Jul 09 03:56:43.107158 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.106942 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ec70533-29e6-48ec-967a-e958f1926405-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsjvn\" (UID: \"3ec70533-29e6-48ec-967a-e958f1926405\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn" Jul 09 03:56:43.107158 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.106962 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c62r8\" (UniqueName: \"kubernetes.io/projected/3ec70533-29e6-48ec-967a-e958f1926405-kube-api-access-c62r8\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsjvn\" (UID: \"3ec70533-29e6-48ec-967a-e958f1926405\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn" Jul 09 03:56:43.107158 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107033 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0341b993-2f81-4a86-a32d-8c26da6b80bb-trusted-ca\") pod \"console-operator-68b757fb67-ln9ht\" (UID: \"0341b993-2f81-4a86-a32d-8c26da6b80bb\") " pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:56:43.107158 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107058 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8ppvs\" (UID: \"75a13467-5b99-4f2f-88fa-7fce616b4560\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:56:43.107158 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107081 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eda93eaa-8da1-4e3c-b104-11f30ea64f25-image-registry-private-configuration\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.107158 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107104 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eda93eaa-8da1-4e3c-b104-11f30ea64f25-installation-pull-secrets\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.107373 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107313 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-s486r\" (UID: \"fc8d4547-833f-48d7-a30b-5c067c926f00\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" Jul 09 03:56:43.107409 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107375 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.107442 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107428 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fc8d4547-833f-48d7-a30b-5c067c926f00-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-s486r\" (UID: \"fc8d4547-833f-48d7-a30b-5c067c926f00\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" Jul 09 03:56:43.107502 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.107444 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 03:56:43.107502 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107458 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vv6q\" (UniqueName: \"kubernetes.io/projected/aa89f5c0-4598-4f98-b76c-0748c8a8920b-kube-api-access-5vv6q\") pod \"volume-data-source-validator-5d456fdbcd-c6pgx\" (UID: \"aa89f5c0-4598-4f98-b76c-0748c8a8920b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c6pgx" Jul 09 03:56:43.107502 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107494 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eda93eaa-8da1-4e3c-b104-11f30ea64f25-ca-trust-extracted\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.107636 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.107511 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert podName:fc8d4547-833f-48d7-a30b-5c067c926f00 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:43.607496505 +0000 UTC m=+38.466367596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-s486r" (UID: "fc8d4547-833f-48d7-a30b-5c067c926f00") : secret "networking-console-plugin-cert" not found Jul 09 03:56:43.107636 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.107519 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 03:56:43.107636 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.107536 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c95f48485-m6qvx: secret "image-registry-tls" not found Jul 09 03:56:43.107636 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107548 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-bound-sa-token\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.107636 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107571 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-7n8mr\" (UID: \"26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr" Jul 09 03:56:43.107636 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107598 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-certificates\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.107636 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.107629 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls podName:eda93eaa-8da1-4e3c-b104-11f30ea64f25 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:43.607612099 +0000 UTC m=+38.466483195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls") pod "image-registry-6c95f48485-m6qvx" (UID: "eda93eaa-8da1-4e3c-b104-11f30ea64f25") : secret "image-registry-tls" not found Jul 09 03:56:43.107979 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107660 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hw4m\" (UniqueName: \"kubernetes.io/projected/26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e-kube-api-access-4hw4m\") pod \"service-ca-operator-5d46d9ffd4-7n8mr\" (UID: \"26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr" Jul 09 03:56:43.107979 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107709 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0341b993-2f81-4a86-a32d-8c26da6b80bb-serving-cert\") pod \"console-operator-68b757fb67-ln9ht\" (UID: \"0341b993-2f81-4a86-a32d-8c26da6b80bb\") " pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:56:43.107979 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107737 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/75a13467-5b99-4f2f-88fa-7fce616b4560-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-8ppvs\" (UID: \"75a13467-5b99-4f2f-88fa-7fce616b4560\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:56:43.107979 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107763 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmzh4\" (UniqueName: \"kubernetes.io/projected/75a13467-5b99-4f2f-88fa-7fce616b4560-kube-api-access-tmzh4\") pod \"cluster-monitoring-operator-64786d9cbb-8ppvs\" (UID: \"75a13467-5b99-4f2f-88fa-7fce616b4560\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:56:43.107979 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107788 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ddrwh\" (UID: \"b9c353a9-991d-42f9-a0f5-bbb80dd30918\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" Jul 09 03:56:43.107979 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.107819 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eda93eaa-8da1-4e3c-b104-11f30ea64f25-ca-trust-extracted\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.108265 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.108111 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-certificates\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.108374 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.108352 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eda93eaa-8da1-4e3c-b104-11f30ea64f25-trusted-ca\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.111091 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.111066 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-7n8mr\" (UID: \"26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr" Jul 09 03:56:43.111179 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.111104 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eda93eaa-8da1-4e3c-b104-11f30ea64f25-installation-pull-secrets\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.111236 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.111214 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eda93eaa-8da1-4e3c-b104-11f30ea64f25-image-registry-private-configuration\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.111418 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.111403 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7955c548df-lznzp"] Jul 09 03:56:43.111481 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.111432 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-9v87h"] Jul 09 03:56:43.111481 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.111476 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn"] Jul 09 03:56:43.111577 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.111501 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh"] Jul 09 03:56:43.111577 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.111503 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j9hq8" Jul 09 03:56:43.111577 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.111529 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j9hq8"] Jul 09 03:56:43.111577 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.111551 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bcrtc"] Jul 09 03:56:43.113684 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.113661 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 03:56:43.113772 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.113695 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 03:56:43.113772 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.113716 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 03:56:43.113900 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.113809 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pxg4s\"" Jul 09 03:56:43.114144 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.114123 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnhb7\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-kube-api-access-mnhb7\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.115747 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.115722 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hw4m\" (UniqueName: \"kubernetes.io/projected/26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e-kube-api-access-4hw4m\") pod \"service-ca-operator-5d46d9ffd4-7n8mr\" (UID: \"26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr" Jul 09 03:56:43.116034 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.116017 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-bound-sa-token\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.116095 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.116049 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vv6q\" (UniqueName: \"kubernetes.io/projected/aa89f5c0-4598-4f98-b76c-0748c8a8920b-kube-api-access-5vv6q\") pod \"volume-data-source-validator-5d456fdbcd-c6pgx\" (UID: \"aa89f5c0-4598-4f98-b76c-0748c8a8920b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c6pgx" Jul 09 03:56:43.117534 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.117517 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fc8d4547-833f-48d7-a30b-5c067c926f00-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-s486r\" (UID: \"fc8d4547-833f-48d7-a30b-5c067c926f00\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" Jul 09 03:56:43.121898 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.121884 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e-config\") pod \"service-ca-operator-5d46d9ffd4-7n8mr\" (UID: \"26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr" Jul 09 03:56:43.208439 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.208414 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/75a13467-5b99-4f2f-88fa-7fce616b4560-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-8ppvs\" (UID: \"75a13467-5b99-4f2f-88fa-7fce616b4560\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:56:43.208439 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.208443 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmzh4\" (UniqueName: \"kubernetes.io/projected/75a13467-5b99-4f2f-88fa-7fce616b4560-kube-api-access-tmzh4\") pod \"cluster-monitoring-operator-64786d9cbb-8ppvs\" (UID: \"75a13467-5b99-4f2f-88fa-7fce616b4560\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:56:43.208623 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.208473 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2b4080d-3736-4237-9b6f-40b058b4a35c-cert\") pod \"ingress-canary-j9hq8\" (UID: \"f2b4080d-3736-4237-9b6f-40b058b4a35c\") " pod="openshift-ingress-canary/ingress-canary-j9hq8" Jul 09 03:56:43.208623 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.208500 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0341b993-2f81-4a86-a32d-8c26da6b80bb-serving-cert\") pod \"console-operator-68b757fb67-ln9ht\" (UID: \"0341b993-2f81-4a86-a32d-8c26da6b80bb\") " pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:56:43.208623 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.208523 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-default-certificate\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:43.208821 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.208648 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977c6161-7495-4f0d-96de-9a13fe980f4c-tmp-dir\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:43.208821 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.208690 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d38cb655-3e43-4c49-8d67-f750a00ea565-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.208821 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.208759 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:43.208821 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.208779 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz2pv\" (UniqueName: \"kubernetes.io/projected/977c6161-7495-4f0d-96de-9a13fe980f4c-kube-api-access-sz2pv\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:43.208821 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.208805 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7nvc\" (UniqueName: \"kubernetes.io/projected/b9c353a9-991d-42f9-a0f5-bbb80dd30918-kube-api-access-d7nvc\") pod \"cluster-samples-operator-56fcf86bf-ddrwh\" (UID: \"b9c353a9-991d-42f9-a0f5-bbb80dd30918\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" Jul 09 03:56:43.209086 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.208835 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0341b993-2f81-4a86-a32d-8c26da6b80bb-trusted-ca\") pod \"console-operator-68b757fb67-ln9ht\" (UID: \"0341b993-2f81-4a86-a32d-8c26da6b80bb\") " pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:56:43.209086 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.208875 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8ppvs\" (UID: \"75a13467-5b99-4f2f-88fa-7fce616b4560\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:56:43.209086 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.208899 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k25jz\" (UniqueName: \"kubernetes.io/projected/5afb8934-2ae6-4904-bea2-6ca59ef52e32-kube-api-access-k25jz\") pod \"network-check-source-989f8b8dc-9v87h\" (UID: \"5afb8934-2ae6-4904-bea2-6ca59ef52e32\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9v87h" Jul 09 03:56:43.209086 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.208917 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgwzq\" (UniqueName: \"kubernetes.io/projected/0341b993-2f81-4a86-a32d-8c26da6b80bb-kube-api-access-kgwzq\") pod \"console-operator-68b757fb67-ln9ht\" (UID: \"0341b993-2f81-4a86-a32d-8c26da6b80bb\") " pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:56:43.209086 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.208996 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 03:56:43.209313 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.209096 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls podName:75a13467-5b99-4f2f-88fa-7fce616b4560 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:43.70907885 +0000 UTC m=+38.567949937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8ppvs" (UID: "75a13467-5b99-4f2f-88fa-7fce616b4560") : secret "cluster-monitoring-operator-tls" not found Jul 09 03:56:43.209313 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209157 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c62r8\" (UniqueName: \"kubernetes.io/projected/3ec70533-29e6-48ec-967a-e958f1926405-kube-api-access-c62r8\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsjvn\" (UID: \"3ec70533-29e6-48ec-967a-e958f1926405\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn" Jul 09 03:56:43.209313 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209190 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:43.209313 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209193 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/75a13467-5b99-4f2f-88fa-7fce616b4560-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-8ppvs\" (UID: \"75a13467-5b99-4f2f-88fa-7fce616b4560\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:56:43.209313 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209217 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d38cb655-3e43-4c49-8d67-f750a00ea565-snapshots\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.209313 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209255 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d38cb655-3e43-4c49-8d67-f750a00ea565-serving-cert\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.209313 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209279 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-stats-auth\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:43.209313 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209303 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:43.209755 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209355 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ddrwh\" (UID: \"b9c353a9-991d-42f9-a0f5-bbb80dd30918\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" Jul 09 03:56:43.209755 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209387 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/977c6161-7495-4f0d-96de-9a13fe980f4c-config-volume\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:43.209755 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209418 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ec70533-29e6-48ec-967a-e958f1926405-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsjvn\" (UID: \"3ec70533-29e6-48ec-967a-e958f1926405\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn" Jul 09 03:56:43.209755 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209457 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxk62\" (UniqueName: \"kubernetes.io/projected/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-kube-api-access-gxk62\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:43.209755 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.209462 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 03:56:43.209755 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209482 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0341b993-2f81-4a86-a32d-8c26da6b80bb-config\") pod \"console-operator-68b757fb67-ln9ht\" (UID: \"0341b993-2f81-4a86-a32d-8c26da6b80bb\") " pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:56:43.209755 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209505 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9586\" (UniqueName: \"kubernetes.io/projected/d38cb655-3e43-4c49-8d67-f750a00ea565-kube-api-access-q9586\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.209755 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.209519 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls podName:b9c353a9-991d-42f9-a0f5-bbb80dd30918 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:43.709503412 +0000 UTC m=+38.568374493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-ddrwh" (UID: "b9c353a9-991d-42f9-a0f5-bbb80dd30918") : secret "samples-operator-tls" not found Jul 09 03:56:43.209755 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209558 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d38cb655-3e43-4c49-8d67-f750a00ea565-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.209755 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209583 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d38cb655-3e43-4c49-8d67-f750a00ea565-tmp\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.209755 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209598 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n988v\" (UniqueName: \"kubernetes.io/projected/f2b4080d-3736-4237-9b6f-40b058b4a35c-kube-api-access-n988v\") pod \"ingress-canary-j9hq8\" (UID: \"f2b4080d-3736-4237-9b6f-40b058b4a35c\") " pod="openshift-ingress-canary/ingress-canary-j9hq8" Jul 09 03:56:43.209755 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209681 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0341b993-2f81-4a86-a32d-8c26da6b80bb-trusted-ca\") pod \"console-operator-68b757fb67-ln9ht\" (UID: \"0341b993-2f81-4a86-a32d-8c26da6b80bb\") " pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:56:43.209755 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209657 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec70533-29e6-48ec-967a-e958f1926405-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsjvn\" (UID: \"3ec70533-29e6-48ec-967a-e958f1926405\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn" Jul 09 03:56:43.210372 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.209979 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ec70533-29e6-48ec-967a-e958f1926405-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsjvn\" (UID: \"3ec70533-29e6-48ec-967a-e958f1926405\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn" Jul 09 03:56:43.210372 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.210025 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0341b993-2f81-4a86-a32d-8c26da6b80bb-config\") pod \"console-operator-68b757fb67-ln9ht\" (UID: \"0341b993-2f81-4a86-a32d-8c26da6b80bb\") " pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:56:43.211122 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.211104 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0341b993-2f81-4a86-a32d-8c26da6b80bb-serving-cert\") pod \"console-operator-68b757fb67-ln9ht\" (UID: \"0341b993-2f81-4a86-a32d-8c26da6b80bb\") " pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:56:43.211772 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.211756 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec70533-29e6-48ec-967a-e958f1926405-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsjvn\" (UID: \"3ec70533-29e6-48ec-967a-e958f1926405\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn" Jul 09 03:56:43.217684 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.217666 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmzh4\" (UniqueName: \"kubernetes.io/projected/75a13467-5b99-4f2f-88fa-7fce616b4560-kube-api-access-tmzh4\") pod \"cluster-monitoring-operator-64786d9cbb-8ppvs\" (UID: \"75a13467-5b99-4f2f-88fa-7fce616b4560\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:56:43.217763 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.217668 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgwzq\" (UniqueName: \"kubernetes.io/projected/0341b993-2f81-4a86-a32d-8c26da6b80bb-kube-api-access-kgwzq\") pod \"console-operator-68b757fb67-ln9ht\" (UID: \"0341b993-2f81-4a86-a32d-8c26da6b80bb\") " pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:56:43.217763 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.217724 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c62r8\" (UniqueName: \"kubernetes.io/projected/3ec70533-29e6-48ec-967a-e958f1926405-kube-api-access-c62r8\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsjvn\" (UID: \"3ec70533-29e6-48ec-967a-e958f1926405\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn" Jul 09 03:56:43.217969 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.217954 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7nvc\" (UniqueName: \"kubernetes.io/projected/b9c353a9-991d-42f9-a0f5-bbb80dd30918-kube-api-access-d7nvc\") pod \"cluster-samples-operator-56fcf86bf-ddrwh\" (UID: \"b9c353a9-991d-42f9-a0f5-bbb80dd30918\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" Jul 09 03:56:43.221454 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.221436 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr" Jul 09 03:56:43.236720 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.236702 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c6pgx" Jul 09 03:56:43.255775 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.255757 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:56:43.276553 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.276510 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn" Jul 09 03:56:43.311178 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.311149 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d38cb655-3e43-4c49-8d67-f750a00ea565-serving-cert\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.311306 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.311184 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-stats-auth\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:43.311306 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.311210 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:43.311448 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.311308 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 03:56:43.311448 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.311400 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs podName:ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:43.811360013 +0000 UTC m=+38.670231106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs") pod "router-default-7955c548df-lznzp" (UID: "ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9") : secret "router-metrics-certs-default" not found Jul 09 03:56:43.311824 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.311802 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/977c6161-7495-4f0d-96de-9a13fe980f4c-config-volume\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:43.311936 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.311837 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxk62\" (UniqueName: \"kubernetes.io/projected/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-kube-api-access-gxk62\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:43.311936 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.311887 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9586\" (UniqueName: \"kubernetes.io/projected/d38cb655-3e43-4c49-8d67-f750a00ea565-kube-api-access-q9586\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.311936 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.311923 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d38cb655-3e43-4c49-8d67-f750a00ea565-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.313741 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.311974 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d38cb655-3e43-4c49-8d67-f750a00ea565-tmp\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.313741 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.312839 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d38cb655-3e43-4c49-8d67-f750a00ea565-tmp\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.313741 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.312923 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n988v\" (UniqueName: \"kubernetes.io/projected/f2b4080d-3736-4237-9b6f-40b058b4a35c-kube-api-access-n988v\") pod \"ingress-canary-j9hq8\" (UID: \"f2b4080d-3736-4237-9b6f-40b058b4a35c\") " pod="openshift-ingress-canary/ingress-canary-j9hq8" Jul 09 03:56:43.313741 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.313004 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2b4080d-3736-4237-9b6f-40b058b4a35c-cert\") pod \"ingress-canary-j9hq8\" (UID: \"f2b4080d-3736-4237-9b6f-40b058b4a35c\") " pod="openshift-ingress-canary/ingress-canary-j9hq8" Jul 09 03:56:43.313741 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.313048 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-default-certificate\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:43.313741 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.313071 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/977c6161-7495-4f0d-96de-9a13fe980f4c-config-volume\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:43.313741 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.313074 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977c6161-7495-4f0d-96de-9a13fe980f4c-tmp-dir\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:43.313741 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.313124 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d38cb655-3e43-4c49-8d67-f750a00ea565-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.313741 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.313151 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:43.313741 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.313180 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz2pv\" (UniqueName: \"kubernetes.io/projected/977c6161-7495-4f0d-96de-9a13fe980f4c-kube-api-access-sz2pv\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:43.313741 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.313233 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k25jz\" (UniqueName: \"kubernetes.io/projected/5afb8934-2ae6-4904-bea2-6ca59ef52e32-kube-api-access-k25jz\") pod \"network-check-source-989f8b8dc-9v87h\" (UID: \"5afb8934-2ae6-4904-bea2-6ca59ef52e32\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9v87h" Jul 09 03:56:43.313741 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.313288 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:43.313741 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.313315 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d38cb655-3e43-4c49-8d67-f750a00ea565-snapshots\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.315273 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.313783 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d38cb655-3e43-4c49-8d67-f750a00ea565-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.315273 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.313799 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d38cb655-3e43-4c49-8d67-f750a00ea565-snapshots\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.315273 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.314220 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle podName:ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:43.814201999 +0000 UTC m=+38.673073095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle") pod "router-default-7955c548df-lznzp" (UID: "ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9") : configmap references non-existent config key: service-ca.crt Jul 09 03:56:43.315273 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.314270 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 03:56:43.315273 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.314317 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls podName:977c6161-7495-4f0d-96de-9a13fe980f4c nodeName:}" failed. No retries permitted until 2026-07-09 03:56:43.814302315 +0000 UTC m=+38.673173408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls") pod "dns-default-bcrtc" (UID: "977c6161-7495-4f0d-96de-9a13fe980f4c") : secret "dns-default-metrics-tls" not found Jul 09 03:56:43.315273 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.314452 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977c6161-7495-4f0d-96de-9a13fe980f4c-tmp-dir\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:43.315273 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.314604 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d38cb655-3e43-4c49-8d67-f750a00ea565-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.327077 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.316499 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-stats-auth\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:43.327077 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.324364 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-default-certificate\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:43.327077 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.326149 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d38cb655-3e43-4c49-8d67-f750a00ea565-serving-cert\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.346319 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.329415 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2b4080d-3736-4237-9b6f-40b058b4a35c-cert\") pod \"ingress-canary-j9hq8\" (UID: \"f2b4080d-3736-4237-9b6f-40b058b4a35c\") " pod="openshift-ingress-canary/ingress-canary-j9hq8" Jul 09 03:56:43.346319 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.329929 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz2pv\" (UniqueName: \"kubernetes.io/projected/977c6161-7495-4f0d-96de-9a13fe980f4c-kube-api-access-sz2pv\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:43.346319 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.331582 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k25jz\" (UniqueName: \"kubernetes.io/projected/5afb8934-2ae6-4904-bea2-6ca59ef52e32-kube-api-access-k25jz\") pod \"network-check-source-989f8b8dc-9v87h\" (UID: \"5afb8934-2ae6-4904-bea2-6ca59ef52e32\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9v87h" Jul 09 03:56:43.346319 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.332397 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n988v\" (UniqueName: \"kubernetes.io/projected/f2b4080d-3736-4237-9b6f-40b058b4a35c-kube-api-access-n988v\") pod \"ingress-canary-j9hq8\" (UID: \"f2b4080d-3736-4237-9b6f-40b058b4a35c\") " pod="openshift-ingress-canary/ingress-canary-j9hq8" Jul 09 03:56:43.346319 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.332888 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxk62\" (UniqueName: \"kubernetes.io/projected/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-kube-api-access-gxk62\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:43.346319 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.338041 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9586\" (UniqueName: \"kubernetes.io/projected/d38cb655-3e43-4c49-8d67-f750a00ea565-kube-api-access-q9586\") pod \"insights-operator-5c847dcfc4-cv9wx\" (UID: \"d38cb655-3e43-4c49-8d67-f750a00ea565\") " pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.346319 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.345965 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" Jul 09 03:56:43.380885 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.378032 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9v87h" Jul 09 03:56:43.432935 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.432666 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j9hq8" Jul 09 03:56:43.467493 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.467265 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c6pgx"] Jul 09 03:56:43.478618 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:43.478587 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa89f5c0_4598_4f98_b76c_0748c8a8920b.slice/crio-0f654111419ea622fc88fb0597962b13708927ae9ed6ce65d79061f96ab63ef2 WatchSource:0}: Error finding container 0f654111419ea622fc88fb0597962b13708927ae9ed6ce65d79061f96ab63ef2: Status 404 returned error can't find the container with id 0f654111419ea622fc88fb0597962b13708927ae9ed6ce65d79061f96ab63ef2 Jul 09 03:56:43.484337 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.481469 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr"] Jul 09 03:56:43.484337 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.483271 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-ln9ht"] Jul 09 03:56:43.496698 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:43.496676 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26dca4d0_e2ec_468a_b0a2_5c6d7bfda07e.slice/crio-da8edda29a97aa41075ce3907c54968be199a4b8a6877e6dfc845c37656adf09 WatchSource:0}: Error finding container da8edda29a97aa41075ce3907c54968be199a4b8a6877e6dfc845c37656adf09: Status 404 returned error can't find the container with id da8edda29a97aa41075ce3907c54968be199a4b8a6877e6dfc845c37656adf09 Jul 09 03:56:43.515559 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.515288 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn"] Jul 09 03:56:43.519076 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:43.519046 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec70533_29e6_48ec_967a_e958f1926405.slice/crio-acc07f2876836cb2a0c760058147895d02e52560511b245619338cc7a432e73b WatchSource:0}: Error finding container acc07f2876836cb2a0c760058147895d02e52560511b245619338cc7a432e73b: Status 404 returned error can't find the container with id acc07f2876836cb2a0c760058147895d02e52560511b245619338cc7a432e73b Jul 09 03:56:43.602513 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.602487 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j9hq8"] Jul 09 03:56:43.605451 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:43.605427 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b4080d_3736_4237_9b6f_40b058b4a35c.slice/crio-2c35776b0ae60639f05d769c9850b1c3503e85bdfe52056e204ee7e55786191d WatchSource:0}: Error finding container 2c35776b0ae60639f05d769c9850b1c3503e85bdfe52056e204ee7e55786191d: Status 404 returned error can't find the container with id 2c35776b0ae60639f05d769c9850b1c3503e85bdfe52056e204ee7e55786191d Jul 09 03:56:43.618472 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.618452 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-s486r\" (UID: \"fc8d4547-833f-48d7-a30b-5c067c926f00\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" Jul 09 03:56:43.618556 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.618485 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:43.618609 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.618590 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 03:56:43.618646 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.618639 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 03:56:43.618677 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.618648 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c95f48485-m6qvx: secret "image-registry-tls" not found Jul 09 03:56:43.618677 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.618650 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert podName:fc8d4547-833f-48d7-a30b-5c067c926f00 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:44.618634568 +0000 UTC m=+39.477505648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-s486r" (UID: "fc8d4547-833f-48d7-a30b-5c067c926f00") : secret "networking-console-plugin-cert" not found Jul 09 03:56:43.618754 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.618681 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls podName:eda93eaa-8da1-4e3c-b104-11f30ea64f25 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:44.618670716 +0000 UTC m=+39.477541794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls") pod "image-registry-6c95f48485-m6qvx" (UID: "eda93eaa-8da1-4e3c-b104-11f30ea64f25") : secret "image-registry-tls" not found Jul 09 03:56:43.719205 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.719183 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ddrwh\" (UID: \"b9c353a9-991d-42f9-a0f5-bbb80dd30918\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" Jul 09 03:56:43.719353 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.719291 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8ppvs\" (UID: \"75a13467-5b99-4f2f-88fa-7fce616b4560\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:56:43.719353 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.719345 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 03:56:43.719422 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.719408 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls podName:b9c353a9-991d-42f9-a0f5-bbb80dd30918 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:44.719391314 +0000 UTC m=+39.578262409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-ddrwh" (UID: "b9c353a9-991d-42f9-a0f5-bbb80dd30918") : secret "samples-operator-tls" not found Jul 09 03:56:43.719459 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.719418 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 03:56:43.719494 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.719461 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls podName:75a13467-5b99-4f2f-88fa-7fce616b4560 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:44.71944919 +0000 UTC m=+39.578320269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8ppvs" (UID: "75a13467-5b99-4f2f-88fa-7fce616b4560") : secret "cluster-monitoring-operator-tls" not found Jul 09 03:56:43.750496 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.750421 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-cv9wx"] Jul 09 03:56:43.752729 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:43.752705 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38cb655_3e43_4c49_8d67_f750a00ea565.slice/crio-dc791741fadbae4f6b2a6df955c822e5291a2182af8f5cb3ad7c8539377d9e53 WatchSource:0}: Error finding container dc791741fadbae4f6b2a6df955c822e5291a2182af8f5cb3ad7c8539377d9e53: Status 404 returned error can't find the container with id dc791741fadbae4f6b2a6df955c822e5291a2182af8f5cb3ad7c8539377d9e53 Jul 09 03:56:43.761898 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.761876 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-9v87h"] Jul 09 03:56:43.764799 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:56:43.764778 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afb8934_2ae6_4904_bea2_6ca59ef52e32.slice/crio-9b92f770c4f30ae90c08a989a43803e89c6dfbf07f07beb89fd051985dc7cf90 WatchSource:0}: Error finding container 9b92f770c4f30ae90c08a989a43803e89c6dfbf07f07beb89fd051985dc7cf90: Status 404 returned error can't find the container with id 9b92f770c4f30ae90c08a989a43803e89c6dfbf07f07beb89fd051985dc7cf90 Jul 09 03:56:43.820213 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.820191 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:43.820312 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.820253 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:43.820312 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.820277 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:43.820400 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.820344 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle podName:ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:44.820326503 +0000 UTC m=+39.679197602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle") pod "router-default-7955c548df-lznzp" (UID: "ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9") : configmap references non-existent config key: service-ca.crt Jul 09 03:56:43.820400 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.820365 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 03:56:43.820400 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.820379 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 03:56:43.820509 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.820405 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs podName:ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:44.820391162 +0000 UTC m=+39.679262260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs") pod "router-default-7955c548df-lznzp" (UID: "ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9") : secret "router-metrics-certs-default" not found Jul 09 03:56:43.820509 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:43.820418 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls podName:977c6161-7495-4f0d-96de-9a13fe980f4c nodeName:}" failed. No retries permitted until 2026-07-09 03:56:44.820412631 +0000 UTC m=+39.679283709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls") pod "dns-default-bcrtc" (UID: "977c6161-7495-4f0d-96de-9a13fe980f4c") : secret "dns-default-metrics-tls" not found Jul 09 03:56:43.824822 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.824800 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn" event={"ID":"3ec70533-29e6-48ec-967a-e958f1926405","Type":"ContainerStarted","Data":"acc07f2876836cb2a0c760058147895d02e52560511b245619338cc7a432e73b"} Jul 09 03:56:43.825841 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.825821 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr" event={"ID":"26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e","Type":"ContainerStarted","Data":"da8edda29a97aa41075ce3907c54968be199a4b8a6877e6dfc845c37656adf09"} Jul 09 03:56:43.826869 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.826838 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9v87h" event={"ID":"5afb8934-2ae6-4904-bea2-6ca59ef52e32","Type":"ContainerStarted","Data":"9b92f770c4f30ae90c08a989a43803e89c6dfbf07f07beb89fd051985dc7cf90"} Jul 09 03:56:43.827965 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.827944 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" event={"ID":"0341b993-2f81-4a86-a32d-8c26da6b80bb","Type":"ContainerStarted","Data":"447b80ddcbe92735883d4a4dac9b2f9953683affd37305e60d4321718039c885"} Jul 09 03:56:43.828892 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.828873 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j9hq8" event={"ID":"f2b4080d-3736-4237-9b6f-40b058b4a35c","Type":"ContainerStarted","Data":"2c35776b0ae60639f05d769c9850b1c3503e85bdfe52056e204ee7e55786191d"} Jul 09 03:56:43.829884 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.829844 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c6pgx" event={"ID":"aa89f5c0-4598-4f98-b76c-0748c8a8920b","Type":"ContainerStarted","Data":"0f654111419ea622fc88fb0597962b13708927ae9ed6ce65d79061f96ab63ef2"} Jul 09 03:56:43.830866 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.830832 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" event={"ID":"d38cb655-3e43-4c49-8d67-f750a00ea565","Type":"ContainerStarted","Data":"dc791741fadbae4f6b2a6df955c822e5291a2182af8f5cb3ad7c8539377d9e53"} Jul 09 03:56:43.833974 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.833954 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bb7z2" event={"ID":"747cde48-17af-4cda-ab13-0a621520741c","Type":"ContainerStarted","Data":"c90a3bb137706e6d9a470cc5a935e8391a16ea84a24282604f7b33eb43ecb41c"} Jul 09 03:56:43.852845 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:43.852777 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bb7z2" podStartSLOduration=7.054227768 podStartE2EDuration="37.852763249s" totalImagesPullingTime="30.798535481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:56:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:43.851574563 +0000 UTC m=+38.710445662" watchObservedRunningTime="2026-07-09 03:56:43.852763249 +0000 UTC m=+38.711634350" Jul 09 03:56:44.620268 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:44.620063 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:56:44.620268 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:44.620103 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:56:44.620268 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:44.620103 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:56:44.625921 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:44.625556 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 03:56:44.625921 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:44.625610 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vmr74\"" Jul 09 03:56:44.625921 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:44.625763 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s6dbl\"" Jul 09 03:56:44.625921 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:44.625794 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 03:56:44.627722 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:44.626841 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-s486r\" (UID: \"fc8d4547-833f-48d7-a30b-5c067c926f00\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" Jul 09 03:56:44.627722 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:44.626900 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:44.627722 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:44.627040 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 03:56:44.627722 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:44.627053 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c95f48485-m6qvx: secret "image-registry-tls" not found Jul 09 03:56:44.627722 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:44.627102 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls podName:eda93eaa-8da1-4e3c-b104-11f30ea64f25 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:46.627085693 +0000 UTC m=+41.485956777 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls") pod "image-registry-6c95f48485-m6qvx" (UID: "eda93eaa-8da1-4e3c-b104-11f30ea64f25") : secret "image-registry-tls" not found Jul 09 03:56:44.627722 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:44.627421 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 03:56:44.627722 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:44.627465 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert podName:fc8d4547-833f-48d7-a30b-5c067c926f00 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:46.627451194 +0000 UTC m=+41.486322274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-s486r" (UID: "fc8d4547-833f-48d7-a30b-5c067c926f00") : secret "networking-console-plugin-cert" not found Jul 09 03:56:44.727366 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:44.727329 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ddrwh\" (UID: \"b9c353a9-991d-42f9-a0f5-bbb80dd30918\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" Jul 09 03:56:44.727535 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:44.727487 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8ppvs\" (UID: \"75a13467-5b99-4f2f-88fa-7fce616b4560\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:56:44.727625 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:44.727610 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 03:56:44.727690 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:44.727676 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls podName:75a13467-5b99-4f2f-88fa-7fce616b4560 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:46.727658041 +0000 UTC m=+41.586529120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8ppvs" (UID: "75a13467-5b99-4f2f-88fa-7fce616b4560") : secret "cluster-monitoring-operator-tls" not found Jul 09 03:56:44.728166 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:44.728146 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 03:56:44.728264 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:44.728207 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls podName:b9c353a9-991d-42f9-a0f5-bbb80dd30918 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:46.728190499 +0000 UTC m=+41.587061576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-ddrwh" (UID: "b9c353a9-991d-42f9-a0f5-bbb80dd30918") : secret "samples-operator-tls" not found Jul 09 03:56:44.828927 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:44.828893 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:44.829369 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:44.828950 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:44.829369 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:44.829092 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:44.829369 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:44.829239 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle podName:ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:46.829216814 +0000 UTC m=+41.688087897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle") pod "router-default-7955c548df-lznzp" (UID: "ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9") : configmap references non-existent config key: service-ca.crt Jul 09 03:56:44.829673 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:44.829652 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 03:56:44.829747 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:44.829711 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls podName:977c6161-7495-4f0d-96de-9a13fe980f4c nodeName:}" failed. No retries permitted until 2026-07-09 03:56:46.829694375 +0000 UTC m=+41.688565455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls") pod "dns-default-bcrtc" (UID: "977c6161-7495-4f0d-96de-9a13fe980f4c") : secret "dns-default-metrics-tls" not found Jul 09 03:56:44.829820 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:44.829804 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 03:56:44.829892 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:44.829847 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs podName:ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:46.829836228 +0000 UTC m=+41.688707310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs") pod "router-default-7955c548df-lznzp" (UID: "ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9") : secret "router-metrics-certs-default" not found Jul 09 03:56:46.648016 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:46.647980 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-s486r\" (UID: \"fc8d4547-833f-48d7-a30b-5c067c926f00\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" Jul 09 03:56:46.648408 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:46.648036 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:46.648408 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:46.648146 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 03:56:46.648408 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:46.648171 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 03:56:46.648408 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:46.648186 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c95f48485-m6qvx: secret "image-registry-tls" not found Jul 09 03:56:46.648408 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:46.648215 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert podName:fc8d4547-833f-48d7-a30b-5c067c926f00 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:50.64819554 +0000 UTC m=+45.507066639 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-s486r" (UID: "fc8d4547-833f-48d7-a30b-5c067c926f00") : secret "networking-console-plugin-cert" not found Jul 09 03:56:46.648408 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:46.648342 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls podName:eda93eaa-8da1-4e3c-b104-11f30ea64f25 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:50.648330046 +0000 UTC m=+45.507201123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls") pod "image-registry-6c95f48485-m6qvx" (UID: "eda93eaa-8da1-4e3c-b104-11f30ea64f25") : secret "image-registry-tls" not found Jul 09 03:56:46.748812 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:46.748776 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8ppvs\" (UID: \"75a13467-5b99-4f2f-88fa-7fce616b4560\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:56:46.748987 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:46.748893 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ddrwh\" (UID: \"b9c353a9-991d-42f9-a0f5-bbb80dd30918\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" Jul 09 03:56:46.748987 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:46.748969 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 03:56:46.749094 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:46.749018 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 03:56:46.749094 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:46.749045 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls podName:75a13467-5b99-4f2f-88fa-7fce616b4560 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:50.749023041 +0000 UTC m=+45.607894142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8ppvs" (UID: "75a13467-5b99-4f2f-88fa-7fce616b4560") : secret "cluster-monitoring-operator-tls" not found Jul 09 03:56:46.749094 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:46.749077 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls podName:b9c353a9-991d-42f9-a0f5-bbb80dd30918 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:50.749056218 +0000 UTC m=+45.607927296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-ddrwh" (UID: "b9c353a9-991d-42f9-a0f5-bbb80dd30918") : secret "samples-operator-tls" not found Jul 09 03:56:46.850164 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:46.850101 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:46.850344 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:46.850302 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle podName:ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:50.850279918 +0000 UTC m=+45.709151015 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle") pod "router-default-7955c548df-lznzp" (UID: "ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9") : configmap references non-existent config key: service-ca.crt Jul 09 03:56:46.850419 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:46.850383 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:46.850474 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:46.850427 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:46.850589 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:46.850571 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 03:56:46.850649 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:46.850599 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 03:56:46.850649 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:46.850633 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs podName:ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:50.850618326 +0000 UTC m=+45.709489426 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs") pod "router-default-7955c548df-lznzp" (UID: "ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9") : secret "router-metrics-certs-default" not found Jul 09 03:56:46.850773 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:46.850657 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls podName:977c6161-7495-4f0d-96de-9a13fe980f4c nodeName:}" failed. No retries permitted until 2026-07-09 03:56:50.850639701 +0000 UTC m=+45.709510799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls") pod "dns-default-bcrtc" (UID: "977c6161-7495-4f0d-96de-9a13fe980f4c") : secret "dns-default-metrics-tls" not found Jul 09 03:56:50.685114 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:50.685070 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-s486r\" (UID: \"fc8d4547-833f-48d7-a30b-5c067c926f00\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" Jul 09 03:56:50.685114 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:50.685123 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:50.685594 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:50.685247 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 03:56:50.685594 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:50.685249 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 03:56:50.685594 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:50.685330 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert podName:fc8d4547-833f-48d7-a30b-5c067c926f00 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:58.68530982 +0000 UTC m=+53.544180917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-s486r" (UID: "fc8d4547-833f-48d7-a30b-5c067c926f00") : secret "networking-console-plugin-cert" not found Jul 09 03:56:50.685594 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:50.685258 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c95f48485-m6qvx: secret "image-registry-tls" not found Jul 09 03:56:50.685594 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:50.685398 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls podName:eda93eaa-8da1-4e3c-b104-11f30ea64f25 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:58.685382975 +0000 UTC m=+53.544254070 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls") pod "image-registry-6c95f48485-m6qvx" (UID: "eda93eaa-8da1-4e3c-b104-11f30ea64f25") : secret "image-registry-tls" not found Jul 09 03:56:50.785658 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:50.785615 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ddrwh\" (UID: \"b9c353a9-991d-42f9-a0f5-bbb80dd30918\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" Jul 09 03:56:50.785843 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:50.785768 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 03:56:50.785843 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:50.785837 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls podName:b9c353a9-991d-42f9-a0f5-bbb80dd30918 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:58.785818223 +0000 UTC m=+53.644689323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-ddrwh" (UID: "b9c353a9-991d-42f9-a0f5-bbb80dd30918") : secret "samples-operator-tls" not found Jul 09 03:56:50.785997 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:50.785770 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8ppvs\" (UID: \"75a13467-5b99-4f2f-88fa-7fce616b4560\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:56:50.785997 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:50.785844 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 03:56:50.786083 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:50.786015 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls podName:75a13467-5b99-4f2f-88fa-7fce616b4560 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:58.785980002 +0000 UTC m=+53.644851079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8ppvs" (UID: "75a13467-5b99-4f2f-88fa-7fce616b4560") : secret "cluster-monitoring-operator-tls" not found Jul 09 03:56:50.886254 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:50.886223 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:50.886414 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:50.886285 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:50.886414 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:50.886320 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:50.886414 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:50.886368 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 03:56:50.886414 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:50.886381 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle podName:ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:58.886365103 +0000 UTC m=+53.745236181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle") pod "router-default-7955c548df-lznzp" (UID: "ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9") : configmap references non-existent config key: service-ca.crt Jul 09 03:56:50.886414 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:50.886414 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls podName:977c6161-7495-4f0d-96de-9a13fe980f4c nodeName:}" failed. No retries permitted until 2026-07-09 03:56:58.886402208 +0000 UTC m=+53.745273285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls") pod "dns-default-bcrtc" (UID: "977c6161-7495-4f0d-96de-9a13fe980f4c") : secret "dns-default-metrics-tls" not found Jul 09 03:56:50.886583 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:50.886426 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 03:56:50.886583 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:50.886460 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs podName:ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:58.886449014 +0000 UTC m=+53.745320096 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs") pod "router-default-7955c548df-lznzp" (UID: "ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9") : secret "router-metrics-certs-default" not found Jul 09 03:56:51.860462 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:51.860043 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" event={"ID":"d38cb655-3e43-4c49-8d67-f750a00ea565","Type":"ContainerStarted","Data":"8cc3f2a312d8acc7121df0b81648008a656300a52bf280c93f06583926f8ce86"} Jul 09 03:56:51.862919 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:51.862887 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn" event={"ID":"3ec70533-29e6-48ec-967a-e958f1926405","Type":"ContainerStarted","Data":"8decbb8fe6e8835c5291dea742835d759cfaa1f02fa841e0b6e52fc57bc403f7"} Jul 09 03:56:51.864665 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:51.864640 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr" event={"ID":"26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e","Type":"ContainerStarted","Data":"4198837b16c1178b479128b9770c4a0abe879c9c2f3ad8d8a2d2745644317c08"} Jul 09 03:56:51.866635 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:51.866608 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9v87h" event={"ID":"5afb8934-2ae6-4904-bea2-6ca59ef52e32","Type":"ContainerStarted","Data":"84856864ea01a85072d060610e48989aba8a823fc0c881a364fa9bae85db227c"} Jul 09 03:56:51.868347 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:51.868327 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/0.log" Jul 09 03:56:51.868440 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:51.868367 2571 generic.go:356] "Generic (PLEG): container finished" podID="0341b993-2f81-4a86-a32d-8c26da6b80bb" containerID="dc004f668a0922a2a44e7c48bdd67af95db06c5bd24b00803440609ee2806bd5" exitCode=255 Jul 09 03:56:51.868497 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:51.868445 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" event={"ID":"0341b993-2f81-4a86-a32d-8c26da6b80bb","Type":"ContainerDied","Data":"dc004f668a0922a2a44e7c48bdd67af95db06c5bd24b00803440609ee2806bd5"} Jul 09 03:56:51.868644 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:51.868618 2571 scope.go:117] "RemoveContainer" containerID="dc004f668a0922a2a44e7c48bdd67af95db06c5bd24b00803440609ee2806bd5" Jul 09 03:56:51.869844 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:51.869824 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j9hq8" event={"ID":"f2b4080d-3736-4237-9b6f-40b058b4a35c","Type":"ContainerStarted","Data":"0f1b2e33dc79775d7364d6b3834ae32a7884cf6edff8813890d5667d44017728"} Jul 09 03:56:51.871354 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:51.871274 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c6pgx" event={"ID":"aa89f5c0-4598-4f98-b76c-0748c8a8920b","Type":"ContainerStarted","Data":"9c50b0d0eebf8cbcfd51b5bebcfc6010b47040301c57c3133f76a061beeee51a"} Jul 09 03:56:51.878140 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:51.878098 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" podStartSLOduration=18.623840931 podStartE2EDuration="25.878083106s" totalImagesPullingTime="7.254242175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:51.876589148 +0000 UTC m=+46.735460248" watchObservedRunningTime="2026-07-09 03:56:51.878083106 +0000 UTC m=+46.736954207" Jul 09 03:56:51.893258 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:51.893129 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn" podStartSLOduration=18.405812595 podStartE2EDuration="25.893110482s" totalImagesPullingTime="7.487297887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:51.892357865 +0000 UTC m=+46.751228966" watchObservedRunningTime="2026-07-09 03:56:51.893110482 +0000 UTC m=+46.751981582" Jul 09 03:56:51.910197 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:51.909949 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr" podStartSLOduration=18.453593613 podStartE2EDuration="25.909929839s" totalImagesPullingTime="7.456336226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:51.909479651 +0000 UTC m=+46.768350753" watchObservedRunningTime="2026-07-09 03:56:51.909929839 +0000 UTC m=+46.768800940" Jul 09 03:56:51.937902 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:51.937820 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c6pgx" podStartSLOduration=18.50626525 podStartE2EDuration="25.937801517s" totalImagesPullingTime="7.431536267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:51.936745854 +0000 UTC m=+46.795616956" watchObservedRunningTime="2026-07-09 03:56:51.937801517 +0000 UTC m=+46.796672618" Jul 09 03:56:51.956698 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:51.956651 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9v87h" podStartSLOduration=18.53215257 podStartE2EDuration="25.95663668s" totalImagesPullingTime="7.42448411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:51.955982799 +0000 UTC m=+46.814853904" watchObservedRunningTime="2026-07-09 03:56:51.95663668 +0000 UTC m=+46.815507780" Jul 09 03:56:52.875984 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:52.875956 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/1.log" Jul 09 03:56:52.876376 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:52.876363 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/0.log" Jul 09 03:56:52.876442 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:52.876395 2571 generic.go:356] "Generic (PLEG): container finished" podID="0341b993-2f81-4a86-a32d-8c26da6b80bb" containerID="0cd026688244b1a26eee631e0629699a2640225003e30dc743e30abb2427ef1c" exitCode=255 Jul 09 03:56:52.876533 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:52.876508 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" event={"ID":"0341b993-2f81-4a86-a32d-8c26da6b80bb","Type":"ContainerDied","Data":"0cd026688244b1a26eee631e0629699a2640225003e30dc743e30abb2427ef1c"} Jul 09 03:56:52.876632 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:52.876617 2571 scope.go:117] "RemoveContainer" containerID="dc004f668a0922a2a44e7c48bdd67af95db06c5bd24b00803440609ee2806bd5" Jul 09 03:56:52.876876 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:52.876838 2571 scope.go:117] "RemoveContainer" containerID="0cd026688244b1a26eee631e0629699a2640225003e30dc743e30abb2427ef1c" Jul 09 03:56:52.877427 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:52.877053 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-ln9ht_openshift-console-operator(0341b993-2f81-4a86-a32d-8c26da6b80bb)\"" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" podUID="0341b993-2f81-4a86-a32d-8c26da6b80bb" Jul 09 03:56:52.892863 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:52.892807 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j9hq8" podStartSLOduration=3.484575712 podStartE2EDuration="10.892790109s" totalImagesPullingTime="7.408214397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:56:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:56:51.972796286 +0000 UTC m=+46.831667387" watchObservedRunningTime="2026-07-09 03:56:52.892790109 +0000 UTC m=+47.751661212" Jul 09 03:56:53.173840 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.173754 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-69dwx"] Jul 09 03:56:53.204969 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.204941 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-69dwx"] Jul 09 03:56:53.205123 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.205102 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:53.207706 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.207682 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 03:56:53.207872 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.207684 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 03:56:53.207872 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.207686 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kwlqq\"" Jul 09 03:56:53.256017 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.255991 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:56:53.256143 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.256035 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:56:53.310675 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.310644 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/266cb396-9250-4419-90ac-2609c2404209-crio-socket\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:53.310832 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.310705 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/266cb396-9250-4419-90ac-2609c2404209-data-volume\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:53.310832 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.310724 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:53.310832 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.310778 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ssl5\" (UniqueName: \"kubernetes.io/projected/266cb396-9250-4419-90ac-2609c2404209-kube-api-access-8ssl5\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:53.310995 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.310880 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:53.411478 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.411445 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ssl5\" (UniqueName: \"kubernetes.io/projected/266cb396-9250-4419-90ac-2609c2404209-kube-api-access-8ssl5\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:53.411667 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.411494 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:53.411667 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.411615 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/266cb396-9250-4419-90ac-2609c2404209-crio-socket\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:53.411772 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:53.411661 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 03:56:53.411772 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.411678 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/266cb396-9250-4419-90ac-2609c2404209-data-volume\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:53.411772 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.411702 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:53.411772 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:53.411741 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm podName:266cb396-9250-4419-90ac-2609c2404209 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:53.911718778 +0000 UTC m=+48.770589876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm") pod "insights-runtime-extractor-69dwx" (UID: "266cb396-9250-4419-90ac-2609c2404209") : configmap "kube-rbac-proxy" not found Jul 09 03:56:53.412007 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:53.411798 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 03:56:53.412007 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:53.411847 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls podName:266cb396-9250-4419-90ac-2609c2404209 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:53.911831453 +0000 UTC m=+48.770702534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69dwx" (UID: "266cb396-9250-4419-90ac-2609c2404209") : secret "insights-runtime-extractor-tls" not found Jul 09 03:56:53.412112 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.412032 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/266cb396-9250-4419-90ac-2609c2404209-crio-socket\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:53.412157 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.412144 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/266cb396-9250-4419-90ac-2609c2404209-data-volume\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:53.420947 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.420923 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ssl5\" (UniqueName: \"kubernetes.io/projected/266cb396-9250-4419-90ac-2609c2404209-kube-api-access-8ssl5\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:53.881113 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.881082 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/1.log" Jul 09 03:56:53.881474 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.881423 2571 scope.go:117] "RemoveContainer" containerID="0cd026688244b1a26eee631e0629699a2640225003e30dc743e30abb2427ef1c" Jul 09 03:56:53.881584 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:53.881566 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-ln9ht_openshift-console-operator(0341b993-2f81-4a86-a32d-8c26da6b80bb)\"" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" podUID="0341b993-2f81-4a86-a32d-8c26da6b80bb" Jul 09 03:56:53.917201 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.917163 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:53.917341 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:53.917240 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 03:56:53.917341 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:53.917295 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm podName:266cb396-9250-4419-90ac-2609c2404209 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:54.917276976 +0000 UTC m=+49.776148053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm") pod "insights-runtime-extractor-69dwx" (UID: "266cb396-9250-4419-90ac-2609c2404209") : configmap "kube-rbac-proxy" not found Jul 09 03:56:53.917456 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:53.917399 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:53.917605 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:53.917580 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 03:56:53.917760 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:53.917646 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls podName:266cb396-9250-4419-90ac-2609c2404209 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:54.917630709 +0000 UTC m=+49.776501811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69dwx" (UID: "266cb396-9250-4419-90ac-2609c2404209") : secret "insights-runtime-extractor-tls" not found Jul 09 03:56:54.884775 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:54.884738 2571 scope.go:117] "RemoveContainer" containerID="0cd026688244b1a26eee631e0629699a2640225003e30dc743e30abb2427ef1c" Jul 09 03:56:54.885257 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:54.884952 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-ln9ht_openshift-console-operator(0341b993-2f81-4a86-a32d-8c26da6b80bb)\"" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" podUID="0341b993-2f81-4a86-a32d-8c26da6b80bb" Jul 09 03:56:54.925019 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:54.924978 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:54.925187 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:54.925080 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 03:56:54.925187 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:54.925146 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm podName:266cb396-9250-4419-90ac-2609c2404209 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:56.925127003 +0000 UTC m=+51.783998087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm") pod "insights-runtime-extractor-69dwx" (UID: "266cb396-9250-4419-90ac-2609c2404209") : configmap "kube-rbac-proxy" not found Jul 09 03:56:54.925305 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:54.925224 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:54.925378 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:54.925362 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 03:56:54.925429 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:54.925414 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls podName:266cb396-9250-4419-90ac-2609c2404209 nodeName:}" failed. No retries permitted until 2026-07-09 03:56:56.92539835 +0000 UTC m=+51.784269448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69dwx" (UID: "266cb396-9250-4419-90ac-2609c2404209") : secret "insights-runtime-extractor-tls" not found Jul 09 03:56:54.970750 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:54.970722 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ltzfl_e23a13f6-edd0-42f2-a03e-ba0b1cf7186d/dns-node-resolver/0.log" Jul 09 03:56:56.170770 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:56.170741 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gn78p_15fb2067-4603-4c89-95d2-92f7f719166a/node-ca/0.log" Jul 09 03:56:56.944902 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:56.944848 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:56.945094 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:56.944981 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 03:56:56.945094 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:56.945019 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:56:56.945094 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:56.945054 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm podName:266cb396-9250-4419-90ac-2609c2404209 nodeName:}" failed. No retries permitted until 2026-07-09 03:57:00.945034956 +0000 UTC m=+55.803906065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm") pod "insights-runtime-extractor-69dwx" (UID: "266cb396-9250-4419-90ac-2609c2404209") : configmap "kube-rbac-proxy" not found Jul 09 03:56:56.945209 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:56.945131 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 03:56:56.945209 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:56.945181 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls podName:266cb396-9250-4419-90ac-2609c2404209 nodeName:}" failed. No retries permitted until 2026-07-09 03:57:00.945166649 +0000 UTC m=+55.804037749 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69dwx" (UID: "266cb396-9250-4419-90ac-2609c2404209") : secret "insights-runtime-extractor-tls" not found Jul 09 03:56:56.971647 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:56.971624 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j9hq8_f2b4080d-3736-4237-9b6f-40b058b4a35c/serve-healthcheck-canary/0.log" Jul 09 03:56:58.763625 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:58.763586 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:56:58.764185 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:58.763739 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 03:56:58.764185 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:58.763759 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c95f48485-m6qvx: secret "image-registry-tls" not found Jul 09 03:56:58.764185 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:58.763809 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls podName:eda93eaa-8da1-4e3c-b104-11f30ea64f25 nodeName:}" failed. No retries permitted until 2026-07-09 03:57:14.763795355 +0000 UTC m=+69.622666462 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls") pod "image-registry-6c95f48485-m6qvx" (UID: "eda93eaa-8da1-4e3c-b104-11f30ea64f25") : secret "image-registry-tls" not found Jul 09 03:56:58.764185 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:58.763827 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-s486r\" (UID: \"fc8d4547-833f-48d7-a30b-5c067c926f00\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" Jul 09 03:56:58.764185 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:58.763954 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 03:56:58.764185 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:58.763998 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert podName:fc8d4547-833f-48d7-a30b-5c067c926f00 nodeName:}" failed. No retries permitted until 2026-07-09 03:57:14.763985857 +0000 UTC m=+69.622856963 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-s486r" (UID: "fc8d4547-833f-48d7-a30b-5c067c926f00") : secret "networking-console-plugin-cert" not found Jul 09 03:56:58.864769 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:58.864737 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8ppvs\" (UID: \"75a13467-5b99-4f2f-88fa-7fce616b4560\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:56:58.864944 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:58.864896 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 03:56:58.864985 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:58.864950 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls podName:75a13467-5b99-4f2f-88fa-7fce616b4560 nodeName:}" failed. No retries permitted until 2026-07-09 03:57:14.864937909 +0000 UTC m=+69.723809055 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8ppvs" (UID: "75a13467-5b99-4f2f-88fa-7fce616b4560") : secret "cluster-monitoring-operator-tls" not found Jul 09 03:56:58.864985 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:58.864900 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ddrwh\" (UID: \"b9c353a9-991d-42f9-a0f5-bbb80dd30918\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" Jul 09 03:56:58.864985 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:58.864966 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 03:56:58.865080 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:58.865024 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls podName:b9c353a9-991d-42f9-a0f5-bbb80dd30918 nodeName:}" failed. No retries permitted until 2026-07-09 03:57:14.865010645 +0000 UTC m=+69.723881746 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-ddrwh" (UID: "b9c353a9-991d-42f9-a0f5-bbb80dd30918") : secret "samples-operator-tls" not found Jul 09 03:56:58.965588 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:58.965553 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:58.965752 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:58.965644 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:56:58.965752 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:56:58.965677 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:56:58.965873 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:58.965758 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle podName:ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9 nodeName:}" failed. No retries permitted until 2026-07-09 03:57:14.965739799 +0000 UTC m=+69.824610877 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle") pod "router-default-7955c548df-lznzp" (UID: "ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9") : configmap references non-existent config key: service-ca.crt Jul 09 03:56:58.965873 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:58.965785 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 03:56:58.965873 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:58.965787 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 03:56:58.965991 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:58.965824 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls podName:977c6161-7495-4f0d-96de-9a13fe980f4c nodeName:}" failed. No retries permitted until 2026-07-09 03:57:14.965814344 +0000 UTC m=+69.824685426 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls") pod "dns-default-bcrtc" (UID: "977c6161-7495-4f0d-96de-9a13fe980f4c") : secret "dns-default-metrics-tls" not found Jul 09 03:56:58.966025 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:56:58.965991 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs podName:ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9 nodeName:}" failed. No retries permitted until 2026-07-09 03:57:14.965962934 +0000 UTC m=+69.824834020 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs") pod "router-default-7955c548df-lznzp" (UID: "ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9") : secret "router-metrics-certs-default" not found Jul 09 03:57:00.982968 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:00.982930 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:57:00.983405 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:00.983038 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:57:00.983405 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:57:00.983089 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 03:57:00.983405 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:57:00.983161 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls podName:266cb396-9250-4419-90ac-2609c2404209 nodeName:}" failed. No retries permitted until 2026-07-09 03:57:08.983147341 +0000 UTC m=+63.842018422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69dwx" (UID: "266cb396-9250-4419-90ac-2609c2404209") : secret "insights-runtime-extractor-tls" not found Jul 09 03:57:00.983405 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:57:00.983174 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 03:57:00.983405 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:57:00.983266 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm podName:266cb396-9250-4419-90ac-2609c2404209 nodeName:}" failed. No retries permitted until 2026-07-09 03:57:08.983245547 +0000 UTC m=+63.842116626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm") pod "insights-runtime-extractor-69dwx" (UID: "266cb396-9250-4419-90ac-2609c2404209") : configmap "kube-rbac-proxy" not found Jul 09 03:57:06.620759 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:06.620713 2571 scope.go:117] "RemoveContainer" containerID="0cd026688244b1a26eee631e0629699a2640225003e30dc743e30abb2427ef1c" Jul 09 03:57:06.921358 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:06.921284 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 03:57:06.921699 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:06.921679 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/1.log" Jul 09 03:57:06.921786 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:06.921721 2571 generic.go:356] "Generic (PLEG): container finished" podID="0341b993-2f81-4a86-a32d-8c26da6b80bb" containerID="9454a301709c407c9bb71e433572601d5e13e29a0657e3906a14031332e36a1e" exitCode=255 Jul 09 03:57:06.921837 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:06.921791 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" event={"ID":"0341b993-2f81-4a86-a32d-8c26da6b80bb","Type":"ContainerDied","Data":"9454a301709c407c9bb71e433572601d5e13e29a0657e3906a14031332e36a1e"} Jul 09 03:57:06.921917 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:06.921846 2571 scope.go:117] "RemoveContainer" containerID="0cd026688244b1a26eee631e0629699a2640225003e30dc743e30abb2427ef1c" Jul 09 03:57:06.922235 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:06.922218 2571 scope.go:117] "RemoveContainer" containerID="9454a301709c407c9bb71e433572601d5e13e29a0657e3906a14031332e36a1e" Jul 09 03:57:06.922411 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:57:06.922392 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-ln9ht_openshift-console-operator(0341b993-2f81-4a86-a32d-8c26da6b80bb)\"" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" podUID="0341b993-2f81-4a86-a32d-8c26da6b80bb" Jul 09 03:57:07.925871 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:07.925829 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 03:57:08.818312 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:08.818287 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwnbk" Jul 09 03:57:09.052464 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:09.052423 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:57:09.052876 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:09.052576 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:57:09.052876 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:57:09.052684 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 03:57:09.052876 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:57:09.052764 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls podName:266cb396-9250-4419-90ac-2609c2404209 nodeName:}" failed. No retries permitted until 2026-07-09 03:57:25.05274298 +0000 UTC m=+79.911614078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69dwx" (UID: "266cb396-9250-4419-90ac-2609c2404209") : secret "insights-runtime-extractor-tls" not found Jul 09 03:57:09.053089 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:09.053066 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/266cb396-9250-4419-90ac-2609c2404209-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:57:11.472018 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.471974 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret\") pod \"global-pull-secret-syncer-28sj8\" (UID: \"0d4983fe-8ee1-40b3-89f6-3508333126b3\") " pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:57:11.472459 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.472143 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrj4\" (UniqueName: \"kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4\") pod \"network-check-target-c7ws6\" (UID: \"087e5e53-db4d-4eed-bad5-fb6949ff07c3\") " pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:57:11.472459 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.472194 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs\") pod \"network-metrics-daemon-fvb66\" (UID: \"579aea83-7020-4eb2-9cc9-eb6318890a3e\") " pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:57:11.474481 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.474457 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 03:57:11.474611 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.474516 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 03:57:11.474667 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.474655 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmrj4\" (UniqueName: \"kubernetes.io/projected/087e5e53-db4d-4eed-bad5-fb6949ff07c3-kube-api-access-xmrj4\") pod \"network-check-target-c7ws6\" (UID: \"087e5e53-db4d-4eed-bad5-fb6949ff07c3\") " pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:57:11.484470 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.484448 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d4983fe-8ee1-40b3-89f6-3508333126b3-original-pull-secret\") pod \"global-pull-secret-syncer-28sj8\" (UID: \"0d4983fe-8ee1-40b3-89f6-3508333126b3\") " pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:57:11.484601 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.484580 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579aea83-7020-4eb2-9cc9-eb6318890a3e-metrics-certs\") pod \"network-metrics-daemon-fvb66\" (UID: \"579aea83-7020-4eb2-9cc9-eb6318890a3e\") " pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:57:11.638821 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.638789 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28sj8" Jul 09 03:57:11.654353 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.654332 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s6dbl\"" Jul 09 03:57:11.662142 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.662122 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvb66" Jul 09 03:57:11.663073 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.663058 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vmr74\"" Jul 09 03:57:11.671826 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.671810 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:57:11.810539 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.810501 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-28sj8"] Jul 09 03:57:11.813930 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:57:11.813899 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4983fe_8ee1_40b3_89f6_3508333126b3.slice/crio-8ba5f3caf9affddbab5a68f48de84ef7167b9e570926c00559b46cd7054d518a WatchSource:0}: Error finding container 8ba5f3caf9affddbab5a68f48de84ef7167b9e570926c00559b46cd7054d518a: Status 404 returned error can't find the container with id 8ba5f3caf9affddbab5a68f48de84ef7167b9e570926c00559b46cd7054d518a Jul 09 03:57:11.833468 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.833447 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fvb66"] Jul 09 03:57:11.836068 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:57:11.836042 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579aea83_7020_4eb2_9cc9_eb6318890a3e.slice/crio-d492a5ac0f1f29704ae2b47a6a5761025e4557a04a5a1301b0122f966e3b79ac WatchSource:0}: Error finding container d492a5ac0f1f29704ae2b47a6a5761025e4557a04a5a1301b0122f966e3b79ac: Status 404 returned error can't find the container with id d492a5ac0f1f29704ae2b47a6a5761025e4557a04a5a1301b0122f966e3b79ac Jul 09 03:57:11.857566 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.857539 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c7ws6"] Jul 09 03:57:11.860109 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:57:11.860083 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087e5e53_db4d_4eed_bad5_fb6949ff07c3.slice/crio-ad9344b60b23e8e9e4a5ce404b2ba883387769ad49ba7ac011ceeaf9be98055a WatchSource:0}: Error finding container ad9344b60b23e8e9e4a5ce404b2ba883387769ad49ba7ac011ceeaf9be98055a: Status 404 returned error can't find the container with id ad9344b60b23e8e9e4a5ce404b2ba883387769ad49ba7ac011ceeaf9be98055a Jul 09 03:57:11.939201 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.939165 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c7ws6" event={"ID":"087e5e53-db4d-4eed-bad5-fb6949ff07c3","Type":"ContainerStarted","Data":"ad9344b60b23e8e9e4a5ce404b2ba883387769ad49ba7ac011ceeaf9be98055a"} Jul 09 03:57:11.940234 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.940207 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fvb66" event={"ID":"579aea83-7020-4eb2-9cc9-eb6318890a3e","Type":"ContainerStarted","Data":"d492a5ac0f1f29704ae2b47a6a5761025e4557a04a5a1301b0122f966e3b79ac"} Jul 09 03:57:11.941264 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:11.941241 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-28sj8" event={"ID":"0d4983fe-8ee1-40b3-89f6-3508333126b3","Type":"ContainerStarted","Data":"8ba5f3caf9affddbab5a68f48de84ef7167b9e570926c00559b46cd7054d518a"} Jul 09 03:57:12.946271 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:12.946229 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c7ws6" event={"ID":"087e5e53-db4d-4eed-bad5-fb6949ff07c3","Type":"ContainerStarted","Data":"827288b8e9c6d74b0a99a3de2da0f9b981c94c3ccd185429e567a1e53cb9e057"} Jul 09 03:57:12.946663 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:12.946371 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:57:12.961660 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:12.961611 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c7ws6" podStartSLOduration=66.961593163 podStartE2EDuration="1m6.961593163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:12.961325556 +0000 UTC m=+67.820196656" watchObservedRunningTime="2026-07-09 03:57:12.961593163 +0000 UTC m=+67.820464263" Jul 09 03:57:13.256712 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:13.256684 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:57:13.256848 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:13.256740 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:57:13.257064 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:13.257048 2571 scope.go:117] "RemoveContainer" containerID="9454a301709c407c9bb71e433572601d5e13e29a0657e3906a14031332e36a1e" Jul 09 03:57:13.257248 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:57:13.257226 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-ln9ht_openshift-console-operator(0341b993-2f81-4a86-a32d-8c26da6b80bb)\"" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" podUID="0341b993-2f81-4a86-a32d-8c26da6b80bb" Jul 09 03:57:13.951462 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:13.951425 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fvb66" event={"ID":"579aea83-7020-4eb2-9cc9-eb6318890a3e","Type":"ContainerStarted","Data":"cbea4eb1673a53b081327b4b5dbcb66e4889a1110218cda15f7495a71b795e14"} Jul 09 03:57:13.951935 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:13.951491 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fvb66" event={"ID":"579aea83-7020-4eb2-9cc9-eb6318890a3e","Type":"ContainerStarted","Data":"68a3d7a60bfb25b6dec2e65c352b1b8bca96befd26383d815fda37fd6838c2e7"} Jul 09 03:57:14.802139 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:14.802098 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-s486r\" (UID: \"fc8d4547-833f-48d7-a30b-5c067c926f00\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" Jul 09 03:57:14.802304 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:14.802150 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:57:14.805123 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:14.805094 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls\") pod \"image-registry-6c95f48485-m6qvx\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:57:14.805281 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:14.805171 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fc8d4547-833f-48d7-a30b-5c067c926f00-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-s486r\" (UID: \"fc8d4547-833f-48d7-a30b-5c067c926f00\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" Jul 09 03:57:14.902869 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:14.902827 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8ppvs\" (UID: \"75a13467-5b99-4f2f-88fa-7fce616b4560\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:57:14.903061 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:14.902934 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ddrwh\" (UID: \"b9c353a9-991d-42f9-a0f5-bbb80dd30918\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" Jul 09 03:57:14.905629 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:14.905607 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9c353a9-991d-42f9-a0f5-bbb80dd30918-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ddrwh\" (UID: \"b9c353a9-991d-42f9-a0f5-bbb80dd30918\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" Jul 09 03:57:14.905736 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:14.905612 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a13467-5b99-4f2f-88fa-7fce616b4560-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8ppvs\" (UID: \"75a13467-5b99-4f2f-88fa-7fce616b4560\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:57:14.979695 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:14.979667 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k8plw\"" Jul 09 03:57:14.987436 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:14.987423 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:57:15.001744 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.001719 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-km22d\"" Jul 09 03:57:15.003560 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.003539 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:57:15.003646 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.003573 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:57:15.003646 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.003632 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:57:15.004269 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.004249 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-service-ca-bundle\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:57:15.005949 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.005905 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/977c6161-7495-4f0d-96de-9a13fe980f4c-metrics-tls\") pod \"dns-default-bcrtc\" (UID: \"977c6161-7495-4f0d-96de-9a13fe980f4c\") " pod="openshift-dns/dns-default-bcrtc" Jul 09 03:57:15.006051 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.005976 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9-metrics-certs\") pod \"router-default-7955c548df-lznzp\" (UID: \"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9\") " pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:57:15.009711 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.009690 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" Jul 09 03:57:15.107114 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.107089 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5hpxl\"" Jul 09 03:57:15.115005 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.114913 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" Jul 09 03:57:15.128602 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.128575 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cdr7m\"" Jul 09 03:57:15.132681 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.132637 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fvb66" podStartSLOduration=67.794239014 podStartE2EDuration="1m9.132614475s" totalImagesPullingTime="1.338375461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:13.966847461 +0000 UTC m=+68.825718562" watchObservedRunningTime="2026-07-09 03:57:15.132614475 +0000 UTC m=+69.991485576" Jul 09 03:57:15.133817 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.133776 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c95f48485-m6qvx"] Jul 09 03:57:15.136683 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.136619 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" Jul 09 03:57:15.136797 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:57:15.136608 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda93eaa_8da1_4e3c_b104_11f30ea64f25.slice/crio-1231fe9abca7dab311c5e5ba36a6cec87c4e0e142630b6046202b0c5644c76ab WatchSource:0}: Error finding container 1231fe9abca7dab311c5e5ba36a6cec87c4e0e142630b6046202b0c5644c76ab: Status 404 returned error can't find the container with id 1231fe9abca7dab311c5e5ba36a6cec87c4e0e142630b6046202b0c5644c76ab Jul 09 03:57:15.150485 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.150451 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-s486r"] Jul 09 03:57:15.154609 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:57:15.154575 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8d4547_833f_48d7_a30b_5c067c926f00.slice/crio-96da50416cbcceebcccd90cadd11187ddbd53f02d1df31674cc39406323c38cb WatchSource:0}: Error finding container 96da50416cbcceebcccd90cadd11187ddbd53f02d1df31674cc39406323c38cb: Status 404 returned error can't find the container with id 96da50416cbcceebcccd90cadd11187ddbd53f02d1df31674cc39406323c38cb Jul 09 03:57:15.158520 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.158480 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kwjkm\"" Jul 09 03:57:15.166439 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.166280 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:57:15.202260 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.202030 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q7njr\"" Jul 09 03:57:15.209335 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.209152 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bcrtc" Jul 09 03:57:15.287250 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.287164 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs"] Jul 09 03:57:15.348175 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.348137 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh"] Jul 09 03:57:15.396637 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.396520 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7955c548df-lznzp"] Jul 09 03:57:15.400497 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:57:15.400469 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae76e9a1_9bf1_44a6_8e53_f94b66f74ef9.slice/crio-7a6ee8b652296ea6534a9f7ee909931cbcdf9d924e9875cd41df744ae001f3fd WatchSource:0}: Error finding container 7a6ee8b652296ea6534a9f7ee909931cbcdf9d924e9875cd41df744ae001f3fd: Status 404 returned error can't find the container with id 7a6ee8b652296ea6534a9f7ee909931cbcdf9d924e9875cd41df744ae001f3fd Jul 09 03:57:15.425106 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.425061 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bcrtc"] Jul 09 03:57:15.430474 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:57:15.430446 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977c6161_7495_4f0d_96de_9a13fe980f4c.slice/crio-f74d5cfd3cf055642435769bd38a87c3d6a27d39d7ca2c1ec954892fed694f01 WatchSource:0}: Error finding container f74d5cfd3cf055642435769bd38a87c3d6a27d39d7ca2c1ec954892fed694f01: Status 404 returned error can't find the container with id f74d5cfd3cf055642435769bd38a87c3d6a27d39d7ca2c1ec954892fed694f01 Jul 09 03:57:15.962096 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.962048 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" event={"ID":"b9c353a9-991d-42f9-a0f5-bbb80dd30918","Type":"ContainerStarted","Data":"aa09e570023ad246df71168a1c76460672ae8f1781be21baeb43059a6cda4717"} Jul 09 03:57:15.964400 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.964359 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" event={"ID":"eda93eaa-8da1-4e3c-b104-11f30ea64f25","Type":"ContainerStarted","Data":"9fd903705d8034cf8b5f6f91c7d62bf9f58e22ae140ad1af4b77f54d4e251147"} Jul 09 03:57:15.964542 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.964410 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" event={"ID":"eda93eaa-8da1-4e3c-b104-11f30ea64f25","Type":"ContainerStarted","Data":"1231fe9abca7dab311c5e5ba36a6cec87c4e0e142630b6046202b0c5644c76ab"} Jul 09 03:57:15.965211 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.965187 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:57:15.966890 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.966845 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" event={"ID":"75a13467-5b99-4f2f-88fa-7fce616b4560","Type":"ContainerStarted","Data":"793968db3312564ddec0ba7715460fc798bfdea91d150a5131c5c6b07986fa83"} Jul 09 03:57:15.968456 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.968431 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" event={"ID":"fc8d4547-833f-48d7-a30b-5c067c926f00","Type":"ContainerStarted","Data":"96da50416cbcceebcccd90cadd11187ddbd53f02d1df31674cc39406323c38cb"} Jul 09 03:57:15.970441 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.970415 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bcrtc" event={"ID":"977c6161-7495-4f0d-96de-9a13fe980f4c","Type":"ContainerStarted","Data":"f74d5cfd3cf055642435769bd38a87c3d6a27d39d7ca2c1ec954892fed694f01"} Jul 09 03:57:15.972687 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.972664 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7955c548df-lznzp" event={"ID":"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9","Type":"ContainerStarted","Data":"f3e65037523a4ce422bffb8ae564478387a5fad75e2a3f9d2f60db4788c2d99e"} Jul 09 03:57:15.972804 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.972705 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7955c548df-lznzp" event={"ID":"ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9","Type":"ContainerStarted","Data":"7a6ee8b652296ea6534a9f7ee909931cbcdf9d924e9875cd41df744ae001f3fd"} Jul 09 03:57:15.992051 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:15.991994 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" podStartSLOduration=69.991973526 podStartE2EDuration="1m9.991973526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:15.990840737 +0000 UTC m=+70.849711838" watchObservedRunningTime="2026-07-09 03:57:15.991973526 +0000 UTC m=+70.850844630" Jul 09 03:57:16.014985 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:16.014524 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7955c548df-lznzp" podStartSLOduration=50.014509035 podStartE2EDuration="50.014509035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:16.013597281 +0000 UTC m=+70.872468391" watchObservedRunningTime="2026-07-09 03:57:16.014509035 +0000 UTC m=+70.873380137" Jul 09 03:57:16.166955 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:16.166871 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:57:16.170024 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:16.170002 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:57:16.977628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:16.977582 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:57:16.979005 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:16.978981 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7955c548df-lznzp" Jul 09 03:57:20.995627 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:20.995593 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" event={"ID":"75a13467-5b99-4f2f-88fa-7fce616b4560","Type":"ContainerStarted","Data":"e361d49076cfcb34cb72b7f0a181ddc32685f4295c03dee420a69b94f279342c"} Jul 09 03:57:20.997054 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:20.997030 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" event={"ID":"fc8d4547-833f-48d7-a30b-5c067c926f00","Type":"ContainerStarted","Data":"f8a57c48f4310b0c542dd21ea7481a998ff5ce6549392c02b63f12af10f42842"} Jul 09 03:57:20.998906 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:20.998881 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bcrtc" event={"ID":"977c6161-7495-4f0d-96de-9a13fe980f4c","Type":"ContainerStarted","Data":"0ba44420001dcb6256daf6dbed7f0dda4ae5967df9f54ff7c69cecc6713aabcc"} Jul 09 03:57:20.999036 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:20.998928 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bcrtc" event={"ID":"977c6161-7495-4f0d-96de-9a13fe980f4c","Type":"ContainerStarted","Data":"a8968a403701303ea5eff156eaf038c1541bdd70e85fe1a5e14709b616b2f1d5"} Jul 09 03:57:20.999036 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:20.998953 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bcrtc" Jul 09 03:57:21.000351 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:21.000323 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-28sj8" event={"ID":"0d4983fe-8ee1-40b3-89f6-3508333126b3","Type":"ContainerStarted","Data":"983bee7f4b0e3f893da92d068d1bbd1b95e00ca27bf43e8bc63c707f1d22c179"} Jul 09 03:57:21.002012 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:21.001990 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" event={"ID":"b9c353a9-991d-42f9-a0f5-bbb80dd30918","Type":"ContainerStarted","Data":"5e4a97edb24d66792ebb1d678941d4042714b2131c1f80b4ff2965d6038c65da"} Jul 09 03:57:21.002112 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:21.002029 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" event={"ID":"b9c353a9-991d-42f9-a0f5-bbb80dd30918","Type":"ContainerStarted","Data":"bc79f8edbee6555767938595d4b9e17956fb5e63cfd7e05c85fd26a59b00daf4"} Jul 09 03:57:21.011221 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:21.011162 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8ppvs" podStartSLOduration=49.799012902 podStartE2EDuration="55.01114662s" totalImagesPullingTime="5.212133718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:21.010746489 +0000 UTC m=+75.869617595" watchObservedRunningTime="2026-07-09 03:57:21.01114662 +0000 UTC m=+75.870017718" Jul 09 03:57:21.024865 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:21.024823 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-s486r" podStartSLOduration=49.653225654 podStartE2EDuration="55.024810234s" totalImagesPullingTime="5.37158458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:21.024394335 +0000 UTC m=+75.883265444" watchObservedRunningTime="2026-07-09 03:57:21.024810234 +0000 UTC m=+75.883681334" Jul 09 03:57:21.044155 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:21.044101 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ddrwh" podStartSLOduration=49.902678662 podStartE2EDuration="55.044082148s" totalImagesPullingTime="5.141403486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:21.0424283 +0000 UTC m=+75.901299400" watchObservedRunningTime="2026-07-09 03:57:21.044082148 +0000 UTC m=+75.902953249" Jul 09 03:57:21.059073 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:21.059030 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bcrtc" podStartSLOduration=33.960791855 podStartE2EDuration="39.05901549s" totalImagesPullingTime="5.098223635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:56:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:21.057673083 +0000 UTC m=+75.916544183" watchObservedRunningTime="2026-07-09 03:57:21.05901549 +0000 UTC m=+75.917886591" Jul 09 03:57:21.071215 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:21.071174 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-28sj8" podStartSLOduration=66.356279308 podStartE2EDuration="1m15.071161623s" totalImagesPullingTime="8.714882315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:21.07044606 +0000 UTC m=+75.929317162" watchObservedRunningTime="2026-07-09 03:57:21.071161623 +0000 UTC m=+75.930032722" Jul 09 03:57:25.086178 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:25.086126 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:57:25.088449 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:25.088427 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/266cb396-9250-4419-90ac-2609c2404209-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69dwx\" (UID: \"266cb396-9250-4419-90ac-2609c2404209\") " pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:57:25.316563 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:25.316533 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kwlqq\"" Jul 09 03:57:25.325151 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:25.325137 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-69dwx" Jul 09 03:57:25.444613 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:25.444583 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-69dwx"] Jul 09 03:57:25.447695 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:57:25.447657 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266cb396_9250_4419_90ac_2609c2404209.slice/crio-ffbf6d8f4e22b18781550feea518a08698007d772f5a76b85b2a8a64cbcc45d7 WatchSource:0}: Error finding container ffbf6d8f4e22b18781550feea518a08698007d772f5a76b85b2a8a64cbcc45d7: Status 404 returned error can't find the container with id ffbf6d8f4e22b18781550feea518a08698007d772f5a76b85b2a8a64cbcc45d7 Jul 09 03:57:26.023148 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:26.023036 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-69dwx" event={"ID":"266cb396-9250-4419-90ac-2609c2404209","Type":"ContainerStarted","Data":"ffbf6d8f4e22b18781550feea518a08698007d772f5a76b85b2a8a64cbcc45d7"} Jul 09 03:57:28.620824 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:28.620658 2571 scope.go:117] "RemoveContainer" containerID="9454a301709c407c9bb71e433572601d5e13e29a0657e3906a14031332e36a1e" Jul 09 03:57:30.044498 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.044463 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-69dwx" event={"ID":"266cb396-9250-4419-90ac-2609c2404209","Type":"ContainerStarted","Data":"4c99db50b0e1c43110e1456a0a34b6f1dfc292e765e570f5c02c86a9cbb62f83"} Jul 09 03:57:30.044895 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.044539 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-69dwx" event={"ID":"266cb396-9250-4419-90ac-2609c2404209","Type":"ContainerStarted","Data":"66905fdf1d976cec5a824e3aa762e97ea35ce69067b21008ecfe419308404b85"} Jul 09 03:57:30.046641 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.046623 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 03:57:30.046776 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.046687 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" event={"ID":"0341b993-2f81-4a86-a32d-8c26da6b80bb","Type":"ContainerStarted","Data":"bbee46e83fcc609f19f980fd359d9da3f7cec1e1ca9641b4ac46cef2535af366"} Jul 09 03:57:30.047017 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.046984 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:57:30.053393 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.053369 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" Jul 09 03:57:30.070056 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.069995 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-ln9ht" podStartSLOduration=56.558321807 podStartE2EDuration="1m4.069976291s" totalImagesPullingTime="7.511654484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:30.069948086 +0000 UTC m=+84.928819187" watchObservedRunningTime="2026-07-09 03:57:30.069976291 +0000 UTC m=+84.928847393" Jul 09 03:57:30.092597 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.092563 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-npd5x"] Jul 09 03:57:30.094933 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.094906 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-npd5x" Jul 09 03:57:30.097417 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.097390 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gn8jv\"" Jul 09 03:57:30.097417 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.097416 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 03:57:30.098269 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.098249 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 03:57:30.111260 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.111234 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-npd5x"] Jul 09 03:57:30.226985 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.226892 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gf2v\" (UniqueName: \"kubernetes.io/projected/60ef45cd-61f2-41fe-b2a0-778e37e50563-kube-api-access-7gf2v\") pod \"downloads-685bfb5f64-npd5x\" (UID: \"60ef45cd-61f2-41fe-b2a0-778e37e50563\") " pod="openshift-console/downloads-685bfb5f64-npd5x" Jul 09 03:57:30.328218 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.328181 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gf2v\" (UniqueName: \"kubernetes.io/projected/60ef45cd-61f2-41fe-b2a0-778e37e50563-kube-api-access-7gf2v\") pod \"downloads-685bfb5f64-npd5x\" (UID: \"60ef45cd-61f2-41fe-b2a0-778e37e50563\") " pod="openshift-console/downloads-685bfb5f64-npd5x" Jul 09 03:57:30.336102 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.336072 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gf2v\" (UniqueName: \"kubernetes.io/projected/60ef45cd-61f2-41fe-b2a0-778e37e50563-kube-api-access-7gf2v\") pod \"downloads-685bfb5f64-npd5x\" (UID: \"60ef45cd-61f2-41fe-b2a0-778e37e50563\") " pod="openshift-console/downloads-685bfb5f64-npd5x" Jul 09 03:57:30.405101 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.405063 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-npd5x" Jul 09 03:57:30.555740 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:30.555702 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-npd5x"] Jul 09 03:57:30.559918 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:57:30.559887 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ef45cd_61f2_41fe_b2a0_778e37e50563.slice/crio-fb9f790309a1e71e4516cef1ce0cbec51b48c331f84c944f62663a19f75456ef WatchSource:0}: Error finding container fb9f790309a1e71e4516cef1ce0cbec51b48c331f84c944f62663a19f75456ef: Status 404 returned error can't find the container with id fb9f790309a1e71e4516cef1ce0cbec51b48c331f84c944f62663a19f75456ef Jul 09 03:57:31.007798 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:31.007769 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bcrtc" Jul 09 03:57:31.053249 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:31.053209 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-npd5x" event={"ID":"60ef45cd-61f2-41fe-b2a0-778e37e50563","Type":"ContainerStarted","Data":"fb9f790309a1e71e4516cef1ce0cbec51b48c331f84c944f62663a19f75456ef"} Jul 09 03:57:33.063314 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:33.063236 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-69dwx" event={"ID":"266cb396-9250-4419-90ac-2609c2404209","Type":"ContainerStarted","Data":"e5b4a10397d1a605faada56a9912ecb997b944181828705ec36a9f9f7013f960"} Jul 09 03:57:33.080281 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:33.080226 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-69dwx" podStartSLOduration=33.038835766 podStartE2EDuration="40.080208033s" totalImagesPullingTime="7.041372267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:56:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:33.079188921 +0000 UTC m=+87.938060022" watchObservedRunningTime="2026-07-09 03:57:33.080208033 +0000 UTC m=+87.939079136" Jul 09 03:57:34.580970 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.580938 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8"] Jul 09 03:57:34.583820 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.583795 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" Jul 09 03:57:34.586932 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.586903 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 09 03:57:34.587082 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.586942 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 09 03:57:34.587082 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.587057 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 03:57:34.587220 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.587197 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bvwj2\"" Jul 09 03:57:34.594349 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.594326 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tkf4f"] Jul 09 03:57:34.597089 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.597069 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.597408 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.597386 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8"] Jul 09 03:57:34.600121 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.600054 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 03:57:34.600245 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.600157 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6gkkq\"" Jul 09 03:57:34.600536 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.600367 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 03:57:34.600536 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.600372 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 03:57:34.602080 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.602051 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf"] Jul 09 03:57:34.604766 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.604742 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.607412 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.607194 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 09 03:57:34.607412 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.607231 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 09 03:57:34.607412 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.607307 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 09 03:57:34.607412 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.607401 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kgzww\"" Jul 09 03:57:34.618018 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.617976 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf"] Jul 09 03:57:34.768478 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.768440 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/efdbdf52-eaeb-4c14-8218-6193304fa117-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.768673 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.768529 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0139cc95-48ef-4dbc-adcc-8d24da56a352-root\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.768673 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.768567 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/644e4ef9-8650-4747-950c-c3c02200e894-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-9wkz8\" (UID: \"644e4ef9-8650-4747-950c-c3c02200e894\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" Jul 09 03:57:34.768673 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.768597 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0139cc95-48ef-4dbc-adcc-8d24da56a352-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.768673 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.768663 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/efdbdf52-eaeb-4c14-8218-6193304fa117-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.768891 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.768698 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzkzq\" (UniqueName: \"kubernetes.io/projected/efdbdf52-eaeb-4c14-8218-6193304fa117-kube-api-access-qzkzq\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.768891 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.768732 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0139cc95-48ef-4dbc-adcc-8d24da56a352-metrics-client-ca\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.768891 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.768757 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/644e4ef9-8650-4747-950c-c3c02200e894-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-9wkz8\" (UID: \"644e4ef9-8650-4747-950c-c3c02200e894\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" Jul 09 03:57:34.768891 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.768810 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/644e4ef9-8650-4747-950c-c3c02200e894-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-9wkz8\" (UID: \"644e4ef9-8650-4747-950c-c3c02200e894\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" Jul 09 03:57:34.768891 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.768846 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzvgj\" (UniqueName: \"kubernetes.io/projected/644e4ef9-8650-4747-950c-c3c02200e894-kube-api-access-xzvgj\") pod \"openshift-state-metrics-54cf96dfcf-9wkz8\" (UID: \"644e4ef9-8650-4747-950c-c3c02200e894\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" Jul 09 03:57:34.769110 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.768919 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0139cc95-48ef-4dbc-adcc-8d24da56a352-node-exporter-textfile\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.769110 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.768946 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0139cc95-48ef-4dbc-adcc-8d24da56a352-node-exporter-accelerators-collector-config\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.769110 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.768970 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efdbdf52-eaeb-4c14-8218-6193304fa117-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.769110 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.769006 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0139cc95-48ef-4dbc-adcc-8d24da56a352-node-exporter-wtmp\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.769110 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.769051 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp8mn\" (UniqueName: \"kubernetes.io/projected/0139cc95-48ef-4dbc-adcc-8d24da56a352-kube-api-access-cp8mn\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.769110 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.769090 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/efdbdf52-eaeb-4c14-8218-6193304fa117-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.769324 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.769138 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0139cc95-48ef-4dbc-adcc-8d24da56a352-sys\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.769324 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.769191 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/efdbdf52-eaeb-4c14-8218-6193304fa117-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.769324 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.769225 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0139cc95-48ef-4dbc-adcc-8d24da56a352-node-exporter-tls\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.869909 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.869870 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0139cc95-48ef-4dbc-adcc-8d24da56a352-node-exporter-wtmp\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.869909 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.869907 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp8mn\" (UniqueName: \"kubernetes.io/projected/0139cc95-48ef-4dbc-adcc-8d24da56a352-kube-api-access-cp8mn\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.870123 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.869930 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/efdbdf52-eaeb-4c14-8218-6193304fa117-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.870123 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.869948 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0139cc95-48ef-4dbc-adcc-8d24da56a352-sys\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.870123 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.869975 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/efdbdf52-eaeb-4c14-8218-6193304fa117-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.870123 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870001 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0139cc95-48ef-4dbc-adcc-8d24da56a352-node-exporter-tls\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.870123 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870035 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0139cc95-48ef-4dbc-adcc-8d24da56a352-node-exporter-wtmp\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.870123 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870058 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0139cc95-48ef-4dbc-adcc-8d24da56a352-sys\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.870123 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870103 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/efdbdf52-eaeb-4c14-8218-6193304fa117-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.870484 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870140 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0139cc95-48ef-4dbc-adcc-8d24da56a352-root\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.870484 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870178 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/644e4ef9-8650-4747-950c-c3c02200e894-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-9wkz8\" (UID: \"644e4ef9-8650-4747-950c-c3c02200e894\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" Jul 09 03:57:34.870484 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870211 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0139cc95-48ef-4dbc-adcc-8d24da56a352-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.870484 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870244 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/efdbdf52-eaeb-4c14-8218-6193304fa117-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.870484 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870269 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzkzq\" (UniqueName: \"kubernetes.io/projected/efdbdf52-eaeb-4c14-8218-6193304fa117-kube-api-access-qzkzq\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.870484 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870284 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0139cc95-48ef-4dbc-adcc-8d24da56a352-root\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.870484 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870299 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0139cc95-48ef-4dbc-adcc-8d24da56a352-metrics-client-ca\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.870484 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870329 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/644e4ef9-8650-4747-950c-c3c02200e894-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-9wkz8\" (UID: \"644e4ef9-8650-4747-950c-c3c02200e894\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" Jul 09 03:57:34.870484 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870367 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/644e4ef9-8650-4747-950c-c3c02200e894-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-9wkz8\" (UID: \"644e4ef9-8650-4747-950c-c3c02200e894\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" Jul 09 03:57:34.870484 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870393 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzvgj\" (UniqueName: \"kubernetes.io/projected/644e4ef9-8650-4747-950c-c3c02200e894-kube-api-access-xzvgj\") pod \"openshift-state-metrics-54cf96dfcf-9wkz8\" (UID: \"644e4ef9-8650-4747-950c-c3c02200e894\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" Jul 09 03:57:34.870484 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870444 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0139cc95-48ef-4dbc-adcc-8d24da56a352-node-exporter-textfile\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.870484 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870470 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0139cc95-48ef-4dbc-adcc-8d24da56a352-node-exporter-accelerators-collector-config\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.871139 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870494 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efdbdf52-eaeb-4c14-8218-6193304fa117-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.871139 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.870497 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/efdbdf52-eaeb-4c14-8218-6193304fa117-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.871312 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.871263 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/644e4ef9-8650-4747-950c-c3c02200e894-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-9wkz8\" (UID: \"644e4ef9-8650-4747-950c-c3c02200e894\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" Jul 09 03:57:34.871312 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.871270 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efdbdf52-eaeb-4c14-8218-6193304fa117-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.871465 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.871314 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/efdbdf52-eaeb-4c14-8218-6193304fa117-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.871465 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:57:34.871370 2571 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 09 03:57:34.871465 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:57:34.871429 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/644e4ef9-8650-4747-950c-c3c02200e894-openshift-state-metrics-tls podName:644e4ef9-8650-4747-950c-c3c02200e894 nodeName:}" failed. No retries permitted until 2026-07-09 03:57:35.371410487 +0000 UTC m=+90.230281584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/644e4ef9-8650-4747-950c-c3c02200e894-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-9wkz8" (UID: "644e4ef9-8650-4747-950c-c3c02200e894") : secret "openshift-state-metrics-tls" not found Jul 09 03:57:34.871829 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.871761 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0139cc95-48ef-4dbc-adcc-8d24da56a352-node-exporter-textfile\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.871829 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.871804 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0139cc95-48ef-4dbc-adcc-8d24da56a352-metrics-client-ca\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.872107 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.872083 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0139cc95-48ef-4dbc-adcc-8d24da56a352-node-exporter-accelerators-collector-config\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.872839 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.872817 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/efdbdf52-eaeb-4c14-8218-6193304fa117-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.872957 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.872901 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0139cc95-48ef-4dbc-adcc-8d24da56a352-node-exporter-tls\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.873002 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.872959 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0139cc95-48ef-4dbc-adcc-8d24da56a352-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.873115 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.873098 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/efdbdf52-eaeb-4c14-8218-6193304fa117-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.873688 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.873666 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/644e4ef9-8650-4747-950c-c3c02200e894-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-9wkz8\" (UID: \"644e4ef9-8650-4747-950c-c3c02200e894\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" Jul 09 03:57:34.878364 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.878336 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzkzq\" (UniqueName: \"kubernetes.io/projected/efdbdf52-eaeb-4c14-8218-6193304fa117-kube-api-access-qzkzq\") pod \"kube-state-metrics-7c9dbb8b7d-s6rnf\" (UID: \"efdbdf52-eaeb-4c14-8218-6193304fa117\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.878446 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.878339 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp8mn\" (UniqueName: \"kubernetes.io/projected/0139cc95-48ef-4dbc-adcc-8d24da56a352-kube-api-access-cp8mn\") pod \"node-exporter-tkf4f\" (UID: \"0139cc95-48ef-4dbc-adcc-8d24da56a352\") " pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.878446 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.878399 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzvgj\" (UniqueName: \"kubernetes.io/projected/644e4ef9-8650-4747-950c-c3c02200e894-kube-api-access-xzvgj\") pod \"openshift-state-metrics-54cf96dfcf-9wkz8\" (UID: \"644e4ef9-8650-4747-950c-c3c02200e894\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" Jul 09 03:57:34.909613 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.909586 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tkf4f" Jul 09 03:57:34.919338 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:34.919312 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" Jul 09 03:57:34.922062 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:57:34.921987 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0139cc95_48ef_4dbc_adcc_8d24da56a352.slice/crio-449053a9fc33384cd95fe43d99c11b7565f5d3b969a42845e3582a3a6a59785f WatchSource:0}: Error finding container 449053a9fc33384cd95fe43d99c11b7565f5d3b969a42845e3582a3a6a59785f: Status 404 returned error can't find the container with id 449053a9fc33384cd95fe43d99c11b7565f5d3b969a42845e3582a3a6a59785f Jul 09 03:57:35.073710 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:35.073630 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tkf4f" event={"ID":"0139cc95-48ef-4dbc-adcc-8d24da56a352","Type":"ContainerStarted","Data":"449053a9fc33384cd95fe43d99c11b7565f5d3b969a42845e3582a3a6a59785f"} Jul 09 03:57:35.075147 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:35.075060 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf"] Jul 09 03:57:35.078539 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:57:35.078509 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdbdf52_eaeb_4c14_8218_6193304fa117.slice/crio-35a294b22790a5e6a1e2364a3f87b5d513ab1ccb33ba2a9c2cb93cae8ae1131e WatchSource:0}: Error finding container 35a294b22790a5e6a1e2364a3f87b5d513ab1ccb33ba2a9c2cb93cae8ae1131e: Status 404 returned error can't find the container with id 35a294b22790a5e6a1e2364a3f87b5d513ab1ccb33ba2a9c2cb93cae8ae1131e Jul 09 03:57:35.373892 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:35.373843 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/644e4ef9-8650-4747-950c-c3c02200e894-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-9wkz8\" (UID: \"644e4ef9-8650-4747-950c-c3c02200e894\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" Jul 09 03:57:35.376605 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:35.376581 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/644e4ef9-8650-4747-950c-c3c02200e894-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-9wkz8\" (UID: \"644e4ef9-8650-4747-950c-c3c02200e894\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" Jul 09 03:57:35.494202 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:35.494159 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" Jul 09 03:57:35.660285 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:35.660220 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8"] Jul 09 03:57:35.663205 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:57:35.663178 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644e4ef9_8650_4747_950c_c3c02200e894.slice/crio-02350571de9fafc828875d52d2ebc57e0b53b138e345a0264c57802907ea9925 WatchSource:0}: Error finding container 02350571de9fafc828875d52d2ebc57e0b53b138e345a0264c57802907ea9925: Status 404 returned error can't find the container with id 02350571de9fafc828875d52d2ebc57e0b53b138e345a0264c57802907ea9925 Jul 09 03:57:36.081651 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:36.081426 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" event={"ID":"644e4ef9-8650-4747-950c-c3c02200e894","Type":"ContainerStarted","Data":"e559ffd55667476d964efab14365b29da257f7527a3606ac4e37ea350ef3ba8b"} Jul 09 03:57:36.081651 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:36.081515 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" event={"ID":"644e4ef9-8650-4747-950c-c3c02200e894","Type":"ContainerStarted","Data":"ed6a0ccb4b9e0ce0f22bcb34973ee215e03eaa6f299f9d7983faf6410609df4e"} Jul 09 03:57:36.081651 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:36.081545 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" event={"ID":"644e4ef9-8650-4747-950c-c3c02200e894","Type":"ContainerStarted","Data":"02350571de9fafc828875d52d2ebc57e0b53b138e345a0264c57802907ea9925"} Jul 09 03:57:36.083461 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:36.083382 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" event={"ID":"efdbdf52-eaeb-4c14-8218-6193304fa117","Type":"ContainerStarted","Data":"35a294b22790a5e6a1e2364a3f87b5d513ab1ccb33ba2a9c2cb93cae8ae1131e"} Jul 09 03:57:37.092444 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:37.092388 2571 generic.go:356] "Generic (PLEG): container finished" podID="0139cc95-48ef-4dbc-adcc-8d24da56a352" containerID="eabd4d6f22f95ccf34bdba2f27b739b5202b2617cd77b28e5179082e3500f938" exitCode=0 Jul 09 03:57:37.092904 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:37.092503 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tkf4f" event={"ID":"0139cc95-48ef-4dbc-adcc-8d24da56a352","Type":"ContainerDied","Data":"eabd4d6f22f95ccf34bdba2f27b739b5202b2617cd77b28e5179082e3500f938"} Jul 09 03:57:37.096878 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:37.096820 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" event={"ID":"efdbdf52-eaeb-4c14-8218-6193304fa117","Type":"ContainerStarted","Data":"5e343f3671537db87a03cc35fbec79d715cab843b06798fa29fd65090e3d897e"} Jul 09 03:57:37.096963 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:37.096897 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" event={"ID":"efdbdf52-eaeb-4c14-8218-6193304fa117","Type":"ContainerStarted","Data":"78111e0830f6d2057f3cf3955e141ab8dbc59ec3155be9035a67cdaf1ad0b21e"} Jul 09 03:57:37.097017 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:37.096928 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" event={"ID":"efdbdf52-eaeb-4c14-8218-6193304fa117","Type":"ContainerStarted","Data":"4645b1bf55d8906b1b08ac20aa1625ee365b953f74a66b6a3eed69e52a88d7e2"} Jul 09 03:57:37.470545 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:37.470435 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s6rnf" podStartSLOduration=1.832368864 podStartE2EDuration="3.470413209s" totalImagesPullingTime="1.638044345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:57:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:37.134108768 +0000 UTC m=+91.992979879" watchObservedRunningTime="2026-07-09 03:57:37.470413209 +0000 UTC m=+92.329284310" Jul 09 03:57:37.471264 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:37.471243 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c95f48485-m6qvx"] Jul 09 03:57:37.478647 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:37.478621 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:57:38.104276 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:38.104240 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tkf4f" event={"ID":"0139cc95-48ef-4dbc-adcc-8d24da56a352","Type":"ContainerStarted","Data":"b0f9503ab31ce1c5607a0b75b81863e3433b1424a578166fbd8b26c2d91d510a"} Jul 09 03:57:38.105031 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:38.104299 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tkf4f" event={"ID":"0139cc95-48ef-4dbc-adcc-8d24da56a352","Type":"ContainerStarted","Data":"7534f9b28c167722dfa26331872409426fbe6fff5d60d7bc5a4f7d103865fd41"} Jul 09 03:57:38.106828 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:38.106794 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" event={"ID":"644e4ef9-8650-4747-950c-c3c02200e894","Type":"ContainerStarted","Data":"a3e92b4ad0ac1afbbe335dc56af790142b714aa7264672e715f4b088a4a81531"} Jul 09 03:57:38.122024 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:38.121959 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tkf4f" podStartSLOduration=2.746659243 podStartE2EDuration="4.121939222s" totalImagesPullingTime="1.298381773s" totalInitContainerRuntime="76.898206ms" isStatefulPod=true podCreationTimestamp="2026-07-09 03:57:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:38.121091637 +0000 UTC m=+92.979962755" watchObservedRunningTime="2026-07-09 03:57:38.121939222 +0000 UTC m=+92.980810324" Jul 09 03:57:38.137628 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:38.137569 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9wkz8" podStartSLOduration=2.388661383 podStartE2EDuration="4.137547684s" totalImagesPullingTime="1.748886301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:57:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:38.137309437 +0000 UTC m=+92.996180538" watchObservedRunningTime="2026-07-09 03:57:38.137547684 +0000 UTC m=+92.996418785" Jul 09 03:57:39.009058 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.009022 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f7d4c9dd-wv5tf"] Jul 09 03:57:39.012530 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.012507 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.015084 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.015062 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 03:57:39.015907 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.015885 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 03:57:39.016001 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.015932 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 03:57:39.016001 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.015979 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 03:57:39.016001 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.015981 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 03:57:39.016154 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.015932 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-v5bpw\"" Jul 09 03:57:39.023339 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.023316 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f7d4c9dd-wv5tf"] Jul 09 03:57:39.105517 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.105484 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk9nv\" (UniqueName: \"kubernetes.io/projected/4b1654ef-9170-41c2-b1fa-daf270c611f7-kube-api-access-dk9nv\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.105994 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.105531 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-serving-cert\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.105994 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.105657 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-config\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.105994 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.105702 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-service-ca\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.105994 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.105749 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-oauth-serving-cert\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.105994 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.105778 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-oauth-config\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.206119 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.206085 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9nv\" (UniqueName: \"kubernetes.io/projected/4b1654ef-9170-41c2-b1fa-daf270c611f7-kube-api-access-dk9nv\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.206295 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.206139 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-serving-cert\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.206495 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.206464 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-config\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.206626 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.206523 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-service-ca\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.206626 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.206575 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-oauth-serving-cert\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.206626 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.206606 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-oauth-config\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.207266 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.207221 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-config\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.207374 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.207281 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-oauth-serving-cert\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.207374 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.207301 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-service-ca\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.209001 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.208974 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-serving-cert\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.209001 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.208989 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-oauth-config\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.213190 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.213172 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk9nv\" (UniqueName: \"kubernetes.io/projected/4b1654ef-9170-41c2-b1fa-daf270c611f7-kube-api-access-dk9nv\") pod \"console-7f7d4c9dd-wv5tf\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.323021 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.322943 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:57:39.474835 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.474808 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f7d4c9dd-wv5tf"] Jul 09 03:57:39.477831 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:57:39.477806 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1654ef_9170_41c2_b1fa_daf270c611f7.slice/crio-50087f4fe39759d175b1db97ed22ec630afaa140e32521cf083d3c81862cad0f WatchSource:0}: Error finding container 50087f4fe39759d175b1db97ed22ec630afaa140e32521cf083d3c81862cad0f: Status 404 returned error can't find the container with id 50087f4fe39759d175b1db97ed22ec630afaa140e32521cf083d3c81862cad0f Jul 09 03:57:39.801426 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.801382 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-58cfb49f5-dhcrh"] Jul 09 03:57:39.806674 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.806652 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.809092 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.809069 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 09 03:57:39.809208 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.809122 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 09 03:57:39.809208 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.809069 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4cbp5\"" Jul 09 03:57:39.809284 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.809252 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 09 03:57:39.809377 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.809360 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 09 03:57:39.809484 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.809470 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 09 03:57:39.810431 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.810407 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-telemeter-client-tls\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.810527 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.810480 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-secret-telemeter-client\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.810527 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.810513 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.810632 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.810567 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.810694 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.810652 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs2zv\" (UniqueName: \"kubernetes.io/projected/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-kube-api-access-rs2zv\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.810747 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.810702 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-metrics-client-ca\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.810747 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.810738 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-federate-client-tls\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.810879 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.810773 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-serving-certs-ca-bundle\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.816042 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.816007 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 09 03:57:39.819464 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.819444 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58cfb49f5-dhcrh"] Jul 09 03:57:39.911362 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.911324 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-metrics-client-ca\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.911679 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.911373 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-federate-client-tls\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.911679 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.911408 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-serving-certs-ca-bundle\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.911679 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.911443 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-telemeter-client-tls\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.911679 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.911488 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-secret-telemeter-client\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.911679 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.911512 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.911679 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.911554 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.911679 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.911594 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs2zv\" (UniqueName: \"kubernetes.io/projected/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-kube-api-access-rs2zv\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.912583 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.912529 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-metrics-client-ca\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.912690 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.912649 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-serving-certs-ca-bundle\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.912751 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.912696 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.915367 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.915330 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-secret-telemeter-client\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.915473 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.915389 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-federate-client-tls\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.915554 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.915538 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.915755 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.915735 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-telemeter-client-tls\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:39.925910 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:39.921161 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs2zv\" (UniqueName: \"kubernetes.io/projected/7c22c7a1-ffe4-4d18-ba4c-7c78f408d889-kube-api-access-rs2zv\") pod \"telemeter-client-58cfb49f5-dhcrh\" (UID: \"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889\") " pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:40.117432 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:40.117394 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7d4c9dd-wv5tf" event={"ID":"4b1654ef-9170-41c2-b1fa-daf270c611f7","Type":"ContainerStarted","Data":"50087f4fe39759d175b1db97ed22ec630afaa140e32521cf083d3c81862cad0f"} Jul 09 03:57:40.119634 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:40.119612 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" Jul 09 03:57:40.279467 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:40.279411 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58cfb49f5-dhcrh"] Jul 09 03:57:40.282939 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:57:40.282909 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c22c7a1_ffe4_4d18_ba4c_7c78f408d889.slice/crio-6ae504f3b82a98c53b259dc4bfb97b26c3cc88c3a71eb4baac5f02bc7ea264f8 WatchSource:0}: Error finding container 6ae504f3b82a98c53b259dc4bfb97b26c3cc88c3a71eb4baac5f02bc7ea264f8: Status 404 returned error can't find the container with id 6ae504f3b82a98c53b259dc4bfb97b26c3cc88c3a71eb4baac5f02bc7ea264f8 Jul 09 03:57:41.125480 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:41.125437 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" event={"ID":"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889","Type":"ContainerStarted","Data":"6ae504f3b82a98c53b259dc4bfb97b26c3cc88c3a71eb4baac5f02bc7ea264f8"} Jul 09 03:57:43.954168 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:43.954140 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c7ws6" Jul 09 03:57:46.991320 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:46.991237 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5644dcb74-g65vs"] Jul 09 03:57:47.069400 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.069357 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5644dcb74-g65vs"] Jul 09 03:57:47.069581 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.069490 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.076818 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.076792 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 03:57:47.181407 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.181372 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-config\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.181601 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.181472 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-trusted-ca-bundle\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.181601 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.181503 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-oauth-serving-cert\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.181601 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.181539 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-serving-cert\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.181746 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.181631 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2pj7\" (UniqueName: \"kubernetes.io/projected/b221a681-2afa-47d5-be0e-ba04c9b16a2c-kube-api-access-d2pj7\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.181746 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.181721 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-oauth-config\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.181817 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.181756 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-service-ca\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.282805 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.282708 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-trusted-ca-bundle\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.282805 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.282751 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-oauth-serving-cert\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.282805 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.282785 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-serving-cert\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.283100 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.282848 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2pj7\" (UniqueName: \"kubernetes.io/projected/b221a681-2afa-47d5-be0e-ba04c9b16a2c-kube-api-access-d2pj7\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.283100 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.282941 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-oauth-config\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.283100 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.282962 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-service-ca\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.283100 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.282991 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-config\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.283636 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.283611 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-oauth-serving-cert\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.283761 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.283731 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-config\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.283848 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.283809 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-service-ca\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.283848 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.283840 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-trusted-ca-bundle\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.285929 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.285908 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-oauth-config\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.286029 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.285955 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-serving-cert\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.291599 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.291575 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2pj7\" (UniqueName: \"kubernetes.io/projected/b221a681-2afa-47d5-be0e-ba04c9b16a2c-kube-api-access-d2pj7\") pod \"console-5644dcb74-g65vs\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:47.381590 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:47.381547 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:51.054328 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:51.054304 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5644dcb74-g65vs"] Jul 09 03:57:51.057119 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:57:51.057092 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb221a681_2afa_47d5_be0e_ba04c9b16a2c.slice/crio-965dee0032dbdef015ad51aafb30046d19060ee3c8ec03a3bf172335f68350e3 WatchSource:0}: Error finding container 965dee0032dbdef015ad51aafb30046d19060ee3c8ec03a3bf172335f68350e3: Status 404 returned error can't find the container with id 965dee0032dbdef015ad51aafb30046d19060ee3c8ec03a3bf172335f68350e3 Jul 09 03:57:51.174047 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:51.174018 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-npd5x" event={"ID":"60ef45cd-61f2-41fe-b2a0-778e37e50563","Type":"ContainerStarted","Data":"fa6ff30167759c6b180d3a34b540cd8a9552b68145029e4fcf1c8599fc8ebf3b"} Jul 09 03:57:51.174279 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:51.174237 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-npd5x" Jul 09 03:57:51.175665 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:51.175623 2571 patch_prober.go:28] interesting pod/downloads-685bfb5f64-npd5x container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.20:8080/\": dial tcp 10.132.0.20:8080: connect: connection refused" start-of-body= Jul 09 03:57:51.175761 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:51.175670 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-685bfb5f64-npd5x" podUID="60ef45cd-61f2-41fe-b2a0-778e37e50563" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.20:8080/\": dial tcp 10.132.0.20:8080: connect: connection refused" Jul 09 03:57:51.175991 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:51.175967 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5644dcb74-g65vs" event={"ID":"b221a681-2afa-47d5-be0e-ba04c9b16a2c","Type":"ContainerStarted","Data":"132f5df976a65a8467b75f447d2980b08c4b46e4cfa1e45f1868b1253d28e99c"} Jul 09 03:57:51.176059 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:51.176013 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5644dcb74-g65vs" event={"ID":"b221a681-2afa-47d5-be0e-ba04c9b16a2c","Type":"ContainerStarted","Data":"965dee0032dbdef015ad51aafb30046d19060ee3c8ec03a3bf172335f68350e3"} Jul 09 03:57:51.177787 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:51.177765 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7d4c9dd-wv5tf" event={"ID":"4b1654ef-9170-41c2-b1fa-daf270c611f7","Type":"ContainerStarted","Data":"faddded11326cd050dfa9764150a9b2f0753d9b54d4f6e4d9b2df1032217c2be"} Jul 09 03:57:51.189341 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:51.189294 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-npd5x" podStartSLOduration=0.795998909 podStartE2EDuration="21.189276611s" totalImagesPullingTime="20.393277702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:51.18815764 +0000 UTC m=+106.047028742" watchObservedRunningTime="2026-07-09 03:57:51.189276611 +0000 UTC m=+106.048147711" Jul 09 03:57:51.212086 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:51.212038 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f7d4c9dd-wv5tf" podStartSLOduration=1.766500732 podStartE2EDuration="13.212023221s" totalImagesPullingTime="11.445522489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:51.210384178 +0000 UTC m=+106.069255277" watchObservedRunningTime="2026-07-09 03:57:51.212023221 +0000 UTC m=+106.070894322" Jul 09 03:57:51.225984 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:51.225832 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5644dcb74-g65vs" podStartSLOduration=5.225816279 podStartE2EDuration="5.225816279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:51.225004002 +0000 UTC m=+106.083875126" watchObservedRunningTime="2026-07-09 03:57:51.225816279 +0000 UTC m=+106.084687380" Jul 09 03:57:52.186364 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:52.186325 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" event={"ID":"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889","Type":"ContainerStarted","Data":"889c1be934871aade2127830fae36da0150a4d09fb187b213c6d4a6323e442b0"} Jul 09 03:57:52.200464 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:52.200433 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-npd5x" Jul 09 03:57:53.193607 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:53.193564 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" event={"ID":"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889","Type":"ContainerStarted","Data":"772f01d02b133f9489e5fe585881d7d8d47f7e2315a490ed6afde11dcaa52251"} Jul 09 03:57:53.194140 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:53.193627 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" event={"ID":"7c22c7a1-ffe4-4d18-ba4c-7c78f408d889","Type":"ContainerStarted","Data":"250bfbfb97672c4e3f88602c9da385c04aac099a5c40859d0ee5cfefc60485c6"} Jul 09 03:57:53.215783 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:53.215625 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-58cfb49f5-dhcrh" podStartSLOduration=1.764770669 podStartE2EDuration="14.215604163s" totalImagesPullingTime="12.450833494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:57:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:57:53.212806027 +0000 UTC m=+108.071677138" watchObservedRunningTime="2026-07-09 03:57:53.215604163 +0000 UTC m=+108.074475266" Jul 09 03:57:56.358745 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:56.358708 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f7d4c9dd-wv5tf"] Jul 09 03:57:57.382485 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:57.382453 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:57.382937 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:57.382558 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:57.388339 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:57.388316 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:58.218265 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:58.218228 2571 generic.go:356] "Generic (PLEG): container finished" podID="d38cb655-3e43-4c49-8d67-f750a00ea565" containerID="8cc3f2a312d8acc7121df0b81648008a656300a52bf280c93f06583926f8ce86" exitCode=0 Jul 09 03:57:58.218480 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:58.218304 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" event={"ID":"d38cb655-3e43-4c49-8d67-f750a00ea565","Type":"ContainerDied","Data":"8cc3f2a312d8acc7121df0b81648008a656300a52bf280c93f06583926f8ce86"} Jul 09 03:57:58.218916 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:58.218895 2571 scope.go:117] "RemoveContainer" containerID="8cc3f2a312d8acc7121df0b81648008a656300a52bf280c93f06583926f8ce86" Jul 09 03:57:58.222973 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:58.222951 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:57:59.226256 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:59.226219 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-cv9wx" event={"ID":"d38cb655-3e43-4c49-8d67-f750a00ea565","Type":"ContainerStarted","Data":"c0e9cd012c997a4f99b3da21880dfe9eb7c1ff2404c01ec0b685260000552528"} Jul 09 03:57:59.323694 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:57:59.323661 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:58:02.502314 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.502270 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" podUID="eda93eaa-8da1-4e3c-b104-11f30ea64f25" containerName="registry" containerID="cri-o://9fd903705d8034cf8b5f6f91c7d62bf9f58e22ae140ad1af4b77f54d4e251147" gracePeriod=30 Jul 09 03:58:02.772228 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.772172 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:58:02.834074 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.834044 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnhb7\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-kube-api-access-mnhb7\") pod \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " Jul 09 03:58:02.834232 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.834085 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-certificates\") pod \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " Jul 09 03:58:02.834232 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.834109 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eda93eaa-8da1-4e3c-b104-11f30ea64f25-image-registry-private-configuration\") pod \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " Jul 09 03:58:02.834232 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.834145 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-bound-sa-token\") pod \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " Jul 09 03:58:02.834232 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.834173 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eda93eaa-8da1-4e3c-b104-11f30ea64f25-trusted-ca\") pod \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " Jul 09 03:58:02.834232 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.834229 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eda93eaa-8da1-4e3c-b104-11f30ea64f25-ca-trust-extracted\") pod \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " Jul 09 03:58:02.834542 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.834257 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls\") pod \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " Jul 09 03:58:02.834542 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.834290 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eda93eaa-8da1-4e3c-b104-11f30ea64f25-installation-pull-secrets\") pod \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\" (UID: \"eda93eaa-8da1-4e3c-b104-11f30ea64f25\") " Jul 09 03:58:02.835069 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.834659 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda93eaa-8da1-4e3c-b104-11f30ea64f25-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "eda93eaa-8da1-4e3c-b104-11f30ea64f25" (UID: "eda93eaa-8da1-4e3c-b104-11f30ea64f25"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 03:58:02.835069 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.835054 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "eda93eaa-8da1-4e3c-b104-11f30ea64f25" (UID: "eda93eaa-8da1-4e3c-b104-11f30ea64f25"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 03:58:02.836758 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.836729 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-kube-api-access-mnhb7" (OuterVolumeSpecName: "kube-api-access-mnhb7") pod "eda93eaa-8da1-4e3c-b104-11f30ea64f25" (UID: "eda93eaa-8da1-4e3c-b104-11f30ea64f25"). InnerVolumeSpecName "kube-api-access-mnhb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 03:58:02.836938 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.836733 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eda93eaa-8da1-4e3c-b104-11f30ea64f25-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "eda93eaa-8da1-4e3c-b104-11f30ea64f25" (UID: "eda93eaa-8da1-4e3c-b104-11f30ea64f25"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 03:58:02.836938 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.836775 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "eda93eaa-8da1-4e3c-b104-11f30ea64f25" (UID: "eda93eaa-8da1-4e3c-b104-11f30ea64f25"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 03:58:02.836938 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.836894 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eda93eaa-8da1-4e3c-b104-11f30ea64f25-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "eda93eaa-8da1-4e3c-b104-11f30ea64f25" (UID: "eda93eaa-8da1-4e3c-b104-11f30ea64f25"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 03:58:02.836938 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.836929 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "eda93eaa-8da1-4e3c-b104-11f30ea64f25" (UID: "eda93eaa-8da1-4e3c-b104-11f30ea64f25"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 03:58:02.842814 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.842792 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda93eaa-8da1-4e3c-b104-11f30ea64f25-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "eda93eaa-8da1-4e3c-b104-11f30ea64f25" (UID: "eda93eaa-8da1-4e3c-b104-11f30ea64f25"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 03:58:02.935150 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.935123 2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-bound-sa-token\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:02.935150 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.935146 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eda93eaa-8da1-4e3c-b104-11f30ea64f25-trusted-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:02.935303 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.935158 2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eda93eaa-8da1-4e3c-b104-11f30ea64f25-ca-trust-extracted\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:02.935303 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.935171 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:02.935303 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.935182 2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eda93eaa-8da1-4e3c-b104-11f30ea64f25-installation-pull-secrets\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:02.935303 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.935193 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnhb7\" (UniqueName: \"kubernetes.io/projected/eda93eaa-8da1-4e3c-b104-11f30ea64f25-kube-api-access-mnhb7\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:02.935303 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.935205 2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eda93eaa-8da1-4e3c-b104-11f30ea64f25-registry-certificates\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:02.935303 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:02.935217 2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eda93eaa-8da1-4e3c-b104-11f30ea64f25-image-registry-private-configuration\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:03.243832 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:03.243797 2571 generic.go:356] "Generic (PLEG): container finished" podID="eda93eaa-8da1-4e3c-b104-11f30ea64f25" containerID="9fd903705d8034cf8b5f6f91c7d62bf9f58e22ae140ad1af4b77f54d4e251147" exitCode=0 Jul 09 03:58:03.243982 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:03.243838 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" Jul 09 03:58:03.243982 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:03.243873 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" event={"ID":"eda93eaa-8da1-4e3c-b104-11f30ea64f25","Type":"ContainerDied","Data":"9fd903705d8034cf8b5f6f91c7d62bf9f58e22ae140ad1af4b77f54d4e251147"} Jul 09 03:58:03.243982 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:03.243914 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c95f48485-m6qvx" event={"ID":"eda93eaa-8da1-4e3c-b104-11f30ea64f25","Type":"ContainerDied","Data":"1231fe9abca7dab311c5e5ba36a6cec87c4e0e142630b6046202b0c5644c76ab"} Jul 09 03:58:03.243982 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:03.243936 2571 scope.go:117] "RemoveContainer" containerID="9fd903705d8034cf8b5f6f91c7d62bf9f58e22ae140ad1af4b77f54d4e251147" Jul 09 03:58:03.253075 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:03.253059 2571 scope.go:117] "RemoveContainer" containerID="9fd903705d8034cf8b5f6f91c7d62bf9f58e22ae140ad1af4b77f54d4e251147" Jul 09 03:58:03.253321 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:58:03.253301 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd903705d8034cf8b5f6f91c7d62bf9f58e22ae140ad1af4b77f54d4e251147\": container with ID starting with 9fd903705d8034cf8b5f6f91c7d62bf9f58e22ae140ad1af4b77f54d4e251147 not found: ID does not exist" containerID="9fd903705d8034cf8b5f6f91c7d62bf9f58e22ae140ad1af4b77f54d4e251147" Jul 09 03:58:03.253379 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:03.253331 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd903705d8034cf8b5f6f91c7d62bf9f58e22ae140ad1af4b77f54d4e251147"} err="failed to get container status \"9fd903705d8034cf8b5f6f91c7d62bf9f58e22ae140ad1af4b77f54d4e251147\": rpc error: code = NotFound desc = could not find container \"9fd903705d8034cf8b5f6f91c7d62bf9f58e22ae140ad1af4b77f54d4e251147\": container with ID starting with 9fd903705d8034cf8b5f6f91c7d62bf9f58e22ae140ad1af4b77f54d4e251147 not found: ID does not exist" Jul 09 03:58:03.263765 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:03.263742 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c95f48485-m6qvx"] Jul 09 03:58:03.267163 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:03.267142 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c95f48485-m6qvx"] Jul 09 03:58:03.625830 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:03.625798 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eda93eaa-8da1-4e3c-b104-11f30ea64f25" path="/var/lib/kubelet/pods/eda93eaa-8da1-4e3c-b104-11f30ea64f25/volumes" Jul 09 03:58:07.730144 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:07.730109 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5644dcb74-g65vs"] Jul 09 03:58:17.298309 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:17.298230 2571 generic.go:356] "Generic (PLEG): container finished" podID="26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e" containerID="4198837b16c1178b479128b9770c4a0abe879c9c2f3ad8d8a2d2745644317c08" exitCode=0 Jul 09 03:58:17.298693 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:17.298305 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr" event={"ID":"26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e","Type":"ContainerDied","Data":"4198837b16c1178b479128b9770c4a0abe879c9c2f3ad8d8a2d2745644317c08"} Jul 09 03:58:17.298693 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:17.298622 2571 scope.go:117] "RemoveContainer" containerID="4198837b16c1178b479128b9770c4a0abe879c9c2f3ad8d8a2d2745644317c08" Jul 09 03:58:18.303746 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:18.303713 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7n8mr" event={"ID":"26dca4d0-e2ec-468a-b0a2-5c6d7bfda07e","Type":"ContainerStarted","Data":"4fbd8dcb809c1ded77cd3a19a79d96bdef5129f525c0fc7880d0832952f6fc3d"} Jul 09 03:58:21.389088 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.389045 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7f7d4c9dd-wv5tf" podUID="4b1654ef-9170-41c2-b1fa-daf270c611f7" containerName="console" containerID="cri-o://faddded11326cd050dfa9764150a9b2f0753d9b54d4f6e4d9b2df1032217c2be" gracePeriod=15 Jul 09 03:58:21.677102 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.677081 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:58:21.796975 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.796931 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-oauth-config\") pod \"4b1654ef-9170-41c2-b1fa-daf270c611f7\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " Jul 09 03:58:21.797119 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.796980 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-serving-cert\") pod \"4b1654ef-9170-41c2-b1fa-daf270c611f7\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " Jul 09 03:58:21.797119 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.797024 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-config\") pod \"4b1654ef-9170-41c2-b1fa-daf270c611f7\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " Jul 09 03:58:21.797119 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.797063 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-service-ca\") pod \"4b1654ef-9170-41c2-b1fa-daf270c611f7\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " Jul 09 03:58:21.797119 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.797081 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-oauth-serving-cert\") pod \"4b1654ef-9170-41c2-b1fa-daf270c611f7\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " Jul 09 03:58:21.797119 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.797103 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk9nv\" (UniqueName: \"kubernetes.io/projected/4b1654ef-9170-41c2-b1fa-daf270c611f7-kube-api-access-dk9nv\") pod \"4b1654ef-9170-41c2-b1fa-daf270c611f7\" (UID: \"4b1654ef-9170-41c2-b1fa-daf270c611f7\") " Jul 09 03:58:21.797565 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.797513 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-service-ca" (OuterVolumeSpecName: "service-ca") pod "4b1654ef-9170-41c2-b1fa-daf270c611f7" (UID: "4b1654ef-9170-41c2-b1fa-daf270c611f7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 03:58:21.797565 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.797522 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-config" (OuterVolumeSpecName: "console-config") pod "4b1654ef-9170-41c2-b1fa-daf270c611f7" (UID: "4b1654ef-9170-41c2-b1fa-daf270c611f7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 03:58:21.797747 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.797560 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4b1654ef-9170-41c2-b1fa-daf270c611f7" (UID: "4b1654ef-9170-41c2-b1fa-daf270c611f7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 03:58:21.799276 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.799240 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4b1654ef-9170-41c2-b1fa-daf270c611f7" (UID: "4b1654ef-9170-41c2-b1fa-daf270c611f7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 03:58:21.799372 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.799288 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1654ef-9170-41c2-b1fa-daf270c611f7-kube-api-access-dk9nv" (OuterVolumeSpecName: "kube-api-access-dk9nv") pod "4b1654ef-9170-41c2-b1fa-daf270c611f7" (UID: "4b1654ef-9170-41c2-b1fa-daf270c611f7"). InnerVolumeSpecName "kube-api-access-dk9nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 03:58:21.799372 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.799331 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4b1654ef-9170-41c2-b1fa-daf270c611f7" (UID: "4b1654ef-9170-41c2-b1fa-daf270c611f7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 03:58:21.898009 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.897941 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-config\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:21.898009 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.897964 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:21.898009 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.897974 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b1654ef-9170-41c2-b1fa-daf270c611f7-oauth-serving-cert\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:21.898009 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.897983 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk9nv\" (UniqueName: \"kubernetes.io/projected/4b1654ef-9170-41c2-b1fa-daf270c611f7-kube-api-access-dk9nv\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:21.898009 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.897992 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-oauth-config\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:21.898009 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:21.898000 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b1654ef-9170-41c2-b1fa-daf270c611f7-console-serving-cert\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:22.321240 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:22.321156 2571 generic.go:356] "Generic (PLEG): container finished" podID="4b1654ef-9170-41c2-b1fa-daf270c611f7" containerID="faddded11326cd050dfa9764150a9b2f0753d9b54d4f6e4d9b2df1032217c2be" exitCode=0 Jul 09 03:58:22.321240 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:22.321200 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7d4c9dd-wv5tf" Jul 09 03:58:22.321486 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:22.321247 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7d4c9dd-wv5tf" event={"ID":"4b1654ef-9170-41c2-b1fa-daf270c611f7","Type":"ContainerDied","Data":"faddded11326cd050dfa9764150a9b2f0753d9b54d4f6e4d9b2df1032217c2be"} Jul 09 03:58:22.321486 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:22.321300 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7d4c9dd-wv5tf" event={"ID":"4b1654ef-9170-41c2-b1fa-daf270c611f7","Type":"ContainerDied","Data":"50087f4fe39759d175b1db97ed22ec630afaa140e32521cf083d3c81862cad0f"} Jul 09 03:58:22.321486 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:22.321328 2571 scope.go:117] "RemoveContainer" containerID="faddded11326cd050dfa9764150a9b2f0753d9b54d4f6e4d9b2df1032217c2be" Jul 09 03:58:22.322891 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:22.322842 2571 generic.go:356] "Generic (PLEG): container finished" podID="3ec70533-29e6-48ec-967a-e958f1926405" containerID="8decbb8fe6e8835c5291dea742835d759cfaa1f02fa841e0b6e52fc57bc403f7" exitCode=0 Jul 09 03:58:22.323000 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:22.322899 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn" event={"ID":"3ec70533-29e6-48ec-967a-e958f1926405","Type":"ContainerDied","Data":"8decbb8fe6e8835c5291dea742835d759cfaa1f02fa841e0b6e52fc57bc403f7"} Jul 09 03:58:22.323283 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:22.323266 2571 scope.go:117] "RemoveContainer" containerID="8decbb8fe6e8835c5291dea742835d759cfaa1f02fa841e0b6e52fc57bc403f7" Jul 09 03:58:22.330373 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:22.330354 2571 scope.go:117] "RemoveContainer" containerID="faddded11326cd050dfa9764150a9b2f0753d9b54d4f6e4d9b2df1032217c2be" Jul 09 03:58:22.330600 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:58:22.330581 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faddded11326cd050dfa9764150a9b2f0753d9b54d4f6e4d9b2df1032217c2be\": container with ID starting with faddded11326cd050dfa9764150a9b2f0753d9b54d4f6e4d9b2df1032217c2be not found: ID does not exist" containerID="faddded11326cd050dfa9764150a9b2f0753d9b54d4f6e4d9b2df1032217c2be" Jul 09 03:58:22.330667 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:22.330610 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faddded11326cd050dfa9764150a9b2f0753d9b54d4f6e4d9b2df1032217c2be"} err="failed to get container status \"faddded11326cd050dfa9764150a9b2f0753d9b54d4f6e4d9b2df1032217c2be\": rpc error: code = NotFound desc = could not find container \"faddded11326cd050dfa9764150a9b2f0753d9b54d4f6e4d9b2df1032217c2be\": container with ID starting with faddded11326cd050dfa9764150a9b2f0753d9b54d4f6e4d9b2df1032217c2be not found: ID does not exist" Jul 09 03:58:22.351845 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:22.351824 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f7d4c9dd-wv5tf"] Jul 09 03:58:22.353103 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:22.353088 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f7d4c9dd-wv5tf"] Jul 09 03:58:23.330217 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:23.330183 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsjvn" event={"ID":"3ec70533-29e6-48ec-967a-e958f1926405","Type":"ContainerStarted","Data":"d5980163fdf77423314c6b1c4c2125c92ebc4ae890b389f3ee1fc1bb1b18ab1f"} Jul 09 03:58:23.624866 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:23.624826 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b1654ef-9170-41c2-b1fa-daf270c611f7" path="/var/lib/kubelet/pods/4b1654ef-9170-41c2-b1fa-daf270c611f7/volumes" Jul 09 03:58:32.751284 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:32.751219 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5644dcb74-g65vs" podUID="b221a681-2afa-47d5-be0e-ba04c9b16a2c" containerName="console" containerID="cri-o://132f5df976a65a8467b75f447d2980b08c4b46e4cfa1e45f1868b1253d28e99c" gracePeriod=15 Jul 09 03:58:33.034986 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.034955 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:58:33.083575 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.083545 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-trusted-ca-bundle\") pod \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " Jul 09 03:58:33.083575 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.083576 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-oauth-config\") pod \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " Jul 09 03:58:33.083848 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.083597 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-serving-cert\") pod \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " Jul 09 03:58:33.083848 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.083616 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2pj7\" (UniqueName: \"kubernetes.io/projected/b221a681-2afa-47d5-be0e-ba04c9b16a2c-kube-api-access-d2pj7\") pod \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " Jul 09 03:58:33.083848 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.083636 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-config\") pod \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " Jul 09 03:58:33.083848 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.083670 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-oauth-serving-cert\") pod \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " Jul 09 03:58:33.083848 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.083706 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-service-ca\") pod \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\" (UID: \"b221a681-2afa-47d5-be0e-ba04c9b16a2c\") " Jul 09 03:58:33.084139 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.084110 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b221a681-2afa-47d5-be0e-ba04c9b16a2c" (UID: "b221a681-2afa-47d5-be0e-ba04c9b16a2c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 03:58:33.084206 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.084183 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-service-ca" (OuterVolumeSpecName: "service-ca") pod "b221a681-2afa-47d5-be0e-ba04c9b16a2c" (UID: "b221a681-2afa-47d5-be0e-ba04c9b16a2c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 03:58:33.084257 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.084198 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b221a681-2afa-47d5-be0e-ba04c9b16a2c" (UID: "b221a681-2afa-47d5-be0e-ba04c9b16a2c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 03:58:33.084329 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.084304 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-config" (OuterVolumeSpecName: "console-config") pod "b221a681-2afa-47d5-be0e-ba04c9b16a2c" (UID: "b221a681-2afa-47d5-be0e-ba04c9b16a2c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 03:58:33.085845 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.085817 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b221a681-2afa-47d5-be0e-ba04c9b16a2c" (UID: "b221a681-2afa-47d5-be0e-ba04c9b16a2c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 03:58:33.085845 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.085867 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b221a681-2afa-47d5-be0e-ba04c9b16a2c" (UID: "b221a681-2afa-47d5-be0e-ba04c9b16a2c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 03:58:33.086018 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.085944 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b221a681-2afa-47d5-be0e-ba04c9b16a2c-kube-api-access-d2pj7" (OuterVolumeSpecName: "kube-api-access-d2pj7") pod "b221a681-2afa-47d5-be0e-ba04c9b16a2c" (UID: "b221a681-2afa-47d5-be0e-ba04c9b16a2c"). InnerVolumeSpecName "kube-api-access-d2pj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 03:58:33.184731 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.184687 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-trusted-ca-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:33.184731 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.184727 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-oauth-config\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:33.184952 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.184743 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-serving-cert\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:33.184952 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.184757 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2pj7\" (UniqueName: \"kubernetes.io/projected/b221a681-2afa-47d5-be0e-ba04c9b16a2c-kube-api-access-d2pj7\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:33.184952 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.184773 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-console-config\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:33.184952 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.184786 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-oauth-serving-cert\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:33.184952 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.184798 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b221a681-2afa-47d5-be0e-ba04c9b16a2c-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:58:33.370609 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.370575 2571 generic.go:356] "Generic (PLEG): container finished" podID="b221a681-2afa-47d5-be0e-ba04c9b16a2c" containerID="132f5df976a65a8467b75f447d2980b08c4b46e4cfa1e45f1868b1253d28e99c" exitCode=0 Jul 09 03:58:33.370797 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.370623 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5644dcb74-g65vs" Jul 09 03:58:33.370797 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.370627 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5644dcb74-g65vs" event={"ID":"b221a681-2afa-47d5-be0e-ba04c9b16a2c","Type":"ContainerDied","Data":"132f5df976a65a8467b75f447d2980b08c4b46e4cfa1e45f1868b1253d28e99c"} Jul 09 03:58:33.370797 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.370744 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5644dcb74-g65vs" event={"ID":"b221a681-2afa-47d5-be0e-ba04c9b16a2c","Type":"ContainerDied","Data":"965dee0032dbdef015ad51aafb30046d19060ee3c8ec03a3bf172335f68350e3"} Jul 09 03:58:33.370797 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.370771 2571 scope.go:117] "RemoveContainer" containerID="132f5df976a65a8467b75f447d2980b08c4b46e4cfa1e45f1868b1253d28e99c" Jul 09 03:58:33.379308 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.379293 2571 scope.go:117] "RemoveContainer" containerID="132f5df976a65a8467b75f447d2980b08c4b46e4cfa1e45f1868b1253d28e99c" Jul 09 03:58:33.379574 ip-10-0-130-55 kubenswrapper[2571]: E0709 03:58:33.379556 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132f5df976a65a8467b75f447d2980b08c4b46e4cfa1e45f1868b1253d28e99c\": container with ID starting with 132f5df976a65a8467b75f447d2980b08c4b46e4cfa1e45f1868b1253d28e99c not found: ID does not exist" containerID="132f5df976a65a8467b75f447d2980b08c4b46e4cfa1e45f1868b1253d28e99c" Jul 09 03:58:33.379616 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.379585 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132f5df976a65a8467b75f447d2980b08c4b46e4cfa1e45f1868b1253d28e99c"} err="failed to get container status \"132f5df976a65a8467b75f447d2980b08c4b46e4cfa1e45f1868b1253d28e99c\": rpc error: code = NotFound desc = could not find container \"132f5df976a65a8467b75f447d2980b08c4b46e4cfa1e45f1868b1253d28e99c\": container with ID starting with 132f5df976a65a8467b75f447d2980b08c4b46e4cfa1e45f1868b1253d28e99c not found: ID does not exist" Jul 09 03:58:33.390057 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.390032 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5644dcb74-g65vs"] Jul 09 03:58:33.395323 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.395299 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5644dcb74-g65vs"] Jul 09 03:58:33.625266 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:58:33.625193 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b221a681-2afa-47d5-be0e-ba04c9b16a2c" path="/var/lib/kubelet/pods/b221a681-2afa-47d5-be0e-ba04c9b16a2c/volumes" Jul 09 03:59:24.984742 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:24.984711 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx"] Jul 09 03:59:24.985206 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:24.985023 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b221a681-2afa-47d5-be0e-ba04c9b16a2c" containerName="console" Jul 09 03:59:24.985206 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:24.985034 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b221a681-2afa-47d5-be0e-ba04c9b16a2c" containerName="console" Jul 09 03:59:24.985206 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:24.985047 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b1654ef-9170-41c2-b1fa-daf270c611f7" containerName="console" Jul 09 03:59:24.985206 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:24.985052 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b1654ef-9170-41c2-b1fa-daf270c611f7" containerName="console" Jul 09 03:59:24.985206 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:24.985069 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eda93eaa-8da1-4e3c-b104-11f30ea64f25" containerName="registry" Jul 09 03:59:24.985206 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:24.985075 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="eda93eaa-8da1-4e3c-b104-11f30ea64f25" containerName="registry" Jul 09 03:59:24.985206 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:24.985125 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="eda93eaa-8da1-4e3c-b104-11f30ea64f25" containerName="registry" Jul 09 03:59:24.985206 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:24.985135 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b1654ef-9170-41c2-b1fa-daf270c611f7" containerName="console" Jul 09 03:59:24.985206 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:24.985141 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b221a681-2afa-47d5-be0e-ba04c9b16a2c" containerName="console" Jul 09 03:59:24.987977 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:24.987959 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" Jul 09 03:59:24.990353 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:24.990337 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 03:59:24.991127 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:24.991107 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-d248r\"" Jul 09 03:59:24.991229 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:24.991112 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 03:59:24.996189 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:24.996168 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx"] Jul 09 03:59:25.110790 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:25.110754 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9277560f-827c-4911-991d-84d6208447bc-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx\" (UID: \"9277560f-827c-4911-991d-84d6208447bc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" Jul 09 03:59:25.110989 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:25.110824 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqxpv\" (UniqueName: \"kubernetes.io/projected/9277560f-827c-4911-991d-84d6208447bc-kube-api-access-wqxpv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx\" (UID: \"9277560f-827c-4911-991d-84d6208447bc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" Jul 09 03:59:25.110989 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:25.110850 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9277560f-827c-4911-991d-84d6208447bc-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx\" (UID: \"9277560f-827c-4911-991d-84d6208447bc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" Jul 09 03:59:25.211338 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:25.211304 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqxpv\" (UniqueName: \"kubernetes.io/projected/9277560f-827c-4911-991d-84d6208447bc-kube-api-access-wqxpv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx\" (UID: \"9277560f-827c-4911-991d-84d6208447bc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" Jul 09 03:59:25.211338 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:25.211344 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9277560f-827c-4911-991d-84d6208447bc-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx\" (UID: \"9277560f-827c-4911-991d-84d6208447bc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" Jul 09 03:59:25.211523 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:25.211386 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9277560f-827c-4911-991d-84d6208447bc-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx\" (UID: \"9277560f-827c-4911-991d-84d6208447bc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" Jul 09 03:59:25.211772 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:25.211755 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9277560f-827c-4911-991d-84d6208447bc-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx\" (UID: \"9277560f-827c-4911-991d-84d6208447bc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" Jul 09 03:59:25.211808 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:25.211772 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9277560f-827c-4911-991d-84d6208447bc-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx\" (UID: \"9277560f-827c-4911-991d-84d6208447bc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" Jul 09 03:59:25.219569 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:25.219542 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqxpv\" (UniqueName: \"kubernetes.io/projected/9277560f-827c-4911-991d-84d6208447bc-kube-api-access-wqxpv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx\" (UID: \"9277560f-827c-4911-991d-84d6208447bc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" Jul 09 03:59:25.297025 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:25.296934 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" Jul 09 03:59:25.425298 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:25.425223 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx"] Jul 09 03:59:25.429234 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:59:25.429193 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9277560f_827c_4911_991d_84d6208447bc.slice/crio-ca83e9d93c538f021ad2df713f0aa2c6fc67def827d997a9c6fcc0538cc690cb WatchSource:0}: Error finding container ca83e9d93c538f021ad2df713f0aa2c6fc67def827d997a9c6fcc0538cc690cb: Status 404 returned error can't find the container with id ca83e9d93c538f021ad2df713f0aa2c6fc67def827d997a9c6fcc0538cc690cb Jul 09 03:59:25.571465 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:25.571382 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" event={"ID":"9277560f-827c-4911-991d-84d6208447bc","Type":"ContainerStarted","Data":"ca83e9d93c538f021ad2df713f0aa2c6fc67def827d997a9c6fcc0538cc690cb"} Jul 09 03:59:30.299265 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.299224 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c964f4cf9-lqrpk"] Jul 09 03:59:30.307966 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.307930 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.311116 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.310897 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 03:59:30.312106 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.311580 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-v5bpw\"" Jul 09 03:59:30.312106 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.311808 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 03:59:30.312106 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.312043 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 03:59:30.312544 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.312361 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 03:59:30.312544 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.312393 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 03:59:30.312544 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.312438 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c964f4cf9-lqrpk"] Jul 09 03:59:30.317337 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.317307 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 03:59:30.346720 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.346689 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-oauth-serving-cert\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.346720 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.346724 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-console-config\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.346946 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.346743 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55a4799d-da99-4ece-a833-393d63d0d898-console-serving-cert\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.346946 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.346786 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-service-ca\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.346946 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.346887 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55a4799d-da99-4ece-a833-393d63d0d898-console-oauth-config\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.346946 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.346904 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-trusted-ca-bundle\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.346946 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.346917 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fmnx\" (UniqueName: \"kubernetes.io/projected/55a4799d-da99-4ece-a833-393d63d0d898-kube-api-access-5fmnx\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.447908 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.447878 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55a4799d-da99-4ece-a833-393d63d0d898-console-oauth-config\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.447908 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.447912 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-trusted-ca-bundle\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.448140 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.447928 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fmnx\" (UniqueName: \"kubernetes.io/projected/55a4799d-da99-4ece-a833-393d63d0d898-kube-api-access-5fmnx\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.448140 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.447968 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-oauth-serving-cert\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.448140 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.447989 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-console-config\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.448140 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.448007 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55a4799d-da99-4ece-a833-393d63d0d898-console-serving-cert\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.448140 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.448026 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-service-ca\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.448810 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.448786 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-oauth-serving-cert\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.448810 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.448802 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-console-config\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.448970 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.448793 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-service-ca\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.448970 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.448925 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-trusted-ca-bundle\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.450365 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.450344 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55a4799d-da99-4ece-a833-393d63d0d898-console-oauth-config\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.450547 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.450530 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55a4799d-da99-4ece-a833-393d63d0d898-console-serving-cert\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.456276 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.456257 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fmnx\" (UniqueName: \"kubernetes.io/projected/55a4799d-da99-4ece-a833-393d63d0d898-kube-api-access-5fmnx\") pod \"console-c964f4cf9-lqrpk\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.623194 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.623155 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:30.770066 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:30.770042 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c964f4cf9-lqrpk"] Jul 09 03:59:30.772183 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:59:30.772158 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a4799d_da99_4ece_a833_393d63d0d898.slice/crio-bed4e1681326b0e8c220aa7135a87774e532d4da0f66192e106a4f6bc4551978 WatchSource:0}: Error finding container bed4e1681326b0e8c220aa7135a87774e532d4da0f66192e106a4f6bc4551978: Status 404 returned error can't find the container with id bed4e1681326b0e8c220aa7135a87774e532d4da0f66192e106a4f6bc4551978 Jul 09 03:59:31.600539 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:31.600496 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c964f4cf9-lqrpk" event={"ID":"55a4799d-da99-4ece-a833-393d63d0d898","Type":"ContainerStarted","Data":"cb3670f8bc86b7cad8bdd5c43fa3e4a643577a0ffd324d082e4a2c7bc75b1005"} Jul 09 03:59:31.600976 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:31.600559 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c964f4cf9-lqrpk" event={"ID":"55a4799d-da99-4ece-a833-393d63d0d898","Type":"ContainerStarted","Data":"bed4e1681326b0e8c220aa7135a87774e532d4da0f66192e106a4f6bc4551978"} Jul 09 03:59:33.612056 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:33.612018 2571 generic.go:356] "Generic (PLEG): container finished" podID="9277560f-827c-4911-991d-84d6208447bc" containerID="b64c1df36fa4e8f13de44cf9ff81d41884c2f24624db3c58dfa63a423090bd92" exitCode=0 Jul 09 03:59:33.612491 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:33.612109 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" event={"ID":"9277560f-827c-4911-991d-84d6208447bc","Type":"ContainerDied","Data":"b64c1df36fa4e8f13de44cf9ff81d41884c2f24624db3c58dfa63a423090bd92"} Jul 09 03:59:33.630245 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:33.630200 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c964f4cf9-lqrpk" podStartSLOduration=3.6301842669999997 podStartE2EDuration="3.630184267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:59:31.616019236 +0000 UTC m=+206.474890336" watchObservedRunningTime="2026-07-09 03:59:33.630184267 +0000 UTC m=+208.489055382" Jul 09 03:59:36.626150 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:36.626115 2571 generic.go:356] "Generic (PLEG): container finished" podID="9277560f-827c-4911-991d-84d6208447bc" containerID="c9a73792f25fa16a2afab8e863c8eb9a690eb970f4b6416170cdc59425c76576" exitCode=0 Jul 09 03:59:36.626522 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:36.626171 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" event={"ID":"9277560f-827c-4911-991d-84d6208447bc","Type":"ContainerDied","Data":"c9a73792f25fa16a2afab8e863c8eb9a690eb970f4b6416170cdc59425c76576"} Jul 09 03:59:40.623663 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:40.623625 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:40.624061 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:40.623689 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:40.628454 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:40.628433 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:40.647276 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:40.647252 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 03:59:45.666287 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:45.666248 2571 generic.go:356] "Generic (PLEG): container finished" podID="9277560f-827c-4911-991d-84d6208447bc" containerID="c09fc92e39c3658128d37fd0fe32b017af0b21077bd88c33bb79738bad357e93" exitCode=0 Jul 09 03:59:45.666655 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:45.666324 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" event={"ID":"9277560f-827c-4911-991d-84d6208447bc","Type":"ContainerDied","Data":"c09fc92e39c3658128d37fd0fe32b017af0b21077bd88c33bb79738bad357e93"} Jul 09 03:59:46.795374 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:46.795349 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" Jul 09 03:59:46.876952 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:46.876922 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqxpv\" (UniqueName: \"kubernetes.io/projected/9277560f-827c-4911-991d-84d6208447bc-kube-api-access-wqxpv\") pod \"9277560f-827c-4911-991d-84d6208447bc\" (UID: \"9277560f-827c-4911-991d-84d6208447bc\") " Jul 09 03:59:46.876952 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:46.876959 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9277560f-827c-4911-991d-84d6208447bc-util\") pod \"9277560f-827c-4911-991d-84d6208447bc\" (UID: \"9277560f-827c-4911-991d-84d6208447bc\") " Jul 09 03:59:46.877152 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:46.877033 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9277560f-827c-4911-991d-84d6208447bc-bundle\") pod \"9277560f-827c-4911-991d-84d6208447bc\" (UID: \"9277560f-827c-4911-991d-84d6208447bc\") " Jul 09 03:59:46.877698 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:46.877671 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9277560f-827c-4911-991d-84d6208447bc-bundle" (OuterVolumeSpecName: "bundle") pod "9277560f-827c-4911-991d-84d6208447bc" (UID: "9277560f-827c-4911-991d-84d6208447bc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 03:59:46.879216 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:46.879190 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9277560f-827c-4911-991d-84d6208447bc-kube-api-access-wqxpv" (OuterVolumeSpecName: "kube-api-access-wqxpv") pod "9277560f-827c-4911-991d-84d6208447bc" (UID: "9277560f-827c-4911-991d-84d6208447bc"). InnerVolumeSpecName "kube-api-access-wqxpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 03:59:46.881007 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:46.880985 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9277560f-827c-4911-991d-84d6208447bc-util" (OuterVolumeSpecName: "util") pod "9277560f-827c-4911-991d-84d6208447bc" (UID: "9277560f-827c-4911-991d-84d6208447bc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 03:59:46.978158 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:46.978084 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9277560f-827c-4911-991d-84d6208447bc-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:59:46.978158 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:46.978110 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqxpv\" (UniqueName: \"kubernetes.io/projected/9277560f-827c-4911-991d-84d6208447bc-kube-api-access-wqxpv\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:59:46.978158 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:46.978123 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9277560f-827c-4911-991d-84d6208447bc-util\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 03:59:47.675913 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:47.675888 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" Jul 09 03:59:47.676060 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:47.675887 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xq2xx" event={"ID":"9277560f-827c-4911-991d-84d6208447bc","Type":"ContainerDied","Data":"ca83e9d93c538f021ad2df713f0aa2c6fc67def827d997a9c6fcc0538cc690cb"} Jul 09 03:59:47.676060 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:47.676001 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca83e9d93c538f021ad2df713f0aa2c6fc67def827d997a9c6fcc0538cc690cb" Jul 09 03:59:52.967361 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:52.967332 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkctn"] Jul 09 03:59:52.967799 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:52.967628 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9277560f-827c-4911-991d-84d6208447bc" containerName="util" Jul 09 03:59:52.967799 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:52.967638 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9277560f-827c-4911-991d-84d6208447bc" containerName="util" Jul 09 03:59:52.967799 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:52.967649 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9277560f-827c-4911-991d-84d6208447bc" containerName="extract" Jul 09 03:59:52.967799 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:52.967654 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9277560f-827c-4911-991d-84d6208447bc" containerName="extract" Jul 09 03:59:52.967799 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:52.967671 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9277560f-827c-4911-991d-84d6208447bc" containerName="pull" Jul 09 03:59:52.967799 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:52.967676 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9277560f-827c-4911-991d-84d6208447bc" containerName="pull" Jul 09 03:59:52.967799 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:52.967731 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9277560f-827c-4911-991d-84d6208447bc" containerName="extract" Jul 09 03:59:52.971462 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:52.971446 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkctn" Jul 09 03:59:52.973500 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:52.973477 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-kcp5v\"" Jul 09 03:59:52.973611 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:52.973535 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 09 03:59:52.973658 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:52.973643 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 09 03:59:52.981879 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:52.981828 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkctn"] Jul 09 03:59:53.031101 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:53.031067 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8gsc\" (UniqueName: \"kubernetes.io/projected/e216acdd-1651-40f5-bb4e-92023f1b96f7-kube-api-access-q8gsc\") pod \"cert-manager-operator-controller-manager-55c444bf86-jkctn\" (UID: \"e216acdd-1651-40f5-bb4e-92023f1b96f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkctn" Jul 09 03:59:53.031257 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:53.031112 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e216acdd-1651-40f5-bb4e-92023f1b96f7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jkctn\" (UID: \"e216acdd-1651-40f5-bb4e-92023f1b96f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkctn" Jul 09 03:59:53.132383 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:53.132340 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8gsc\" (UniqueName: \"kubernetes.io/projected/e216acdd-1651-40f5-bb4e-92023f1b96f7-kube-api-access-q8gsc\") pod \"cert-manager-operator-controller-manager-55c444bf86-jkctn\" (UID: \"e216acdd-1651-40f5-bb4e-92023f1b96f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkctn" Jul 09 03:59:53.132615 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:53.132393 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e216acdd-1651-40f5-bb4e-92023f1b96f7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jkctn\" (UID: \"e216acdd-1651-40f5-bb4e-92023f1b96f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkctn" Jul 09 03:59:53.132758 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:53.132743 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e216acdd-1651-40f5-bb4e-92023f1b96f7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jkctn\" (UID: \"e216acdd-1651-40f5-bb4e-92023f1b96f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkctn" Jul 09 03:59:53.139987 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:53.139956 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8gsc\" (UniqueName: \"kubernetes.io/projected/e216acdd-1651-40f5-bb4e-92023f1b96f7-kube-api-access-q8gsc\") pod \"cert-manager-operator-controller-manager-55c444bf86-jkctn\" (UID: \"e216acdd-1651-40f5-bb4e-92023f1b96f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkctn" Jul 09 03:59:53.281273 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:53.281193 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkctn" Jul 09 03:59:53.410605 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:53.410576 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkctn"] Jul 09 03:59:53.413199 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:59:53.413173 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode216acdd_1651_40f5_bb4e_92023f1b96f7.slice/crio-10c7f1efb69c4f84b12b58f39ac8e0a422e716aa782d16a9fea88f4d671b8f64 WatchSource:0}: Error finding container 10c7f1efb69c4f84b12b58f39ac8e0a422e716aa782d16a9fea88f4d671b8f64: Status 404 returned error can't find the container with id 10c7f1efb69c4f84b12b58f39ac8e0a422e716aa782d16a9fea88f4d671b8f64 Jul 09 03:59:53.700563 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:53.700529 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkctn" event={"ID":"e216acdd-1651-40f5-bb4e-92023f1b96f7","Type":"ContainerStarted","Data":"10c7f1efb69c4f84b12b58f39ac8e0a422e716aa782d16a9fea88f4d671b8f64"} Jul 09 03:59:55.711145 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:55.711112 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkctn" event={"ID":"e216acdd-1651-40f5-bb4e-92023f1b96f7","Type":"ContainerStarted","Data":"a90465b1d80c82a64586e20664d0657e84bcb9b08cb6181291788555a48a84ac"} Jul 09 03:59:55.731007 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:55.730952 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkctn" podStartSLOduration=2.052070093 podStartE2EDuration="3.730929141s" totalImagesPullingTime="1.678859048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:59:55.728972738 +0000 UTC m=+230.587843832" watchObservedRunningTime="2026-07-09 03:59:55.730929141 +0000 UTC m=+230.589800253" Jul 09 03:59:56.620987 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.620954 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2"] Jul 09 03:59:56.624247 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.624228 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" Jul 09 03:59:56.630586 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.630557 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 03:59:56.630702 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.630591 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-d248r\"" Jul 09 03:59:56.631122 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.631099 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 03:59:56.634025 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.634002 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2"] Jul 09 03:59:56.763776 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.763737 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c16341e-9824-4abb-af4d-1fa4d37acc1f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2\" (UID: \"3c16341e-9824-4abb-af4d-1fa4d37acc1f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" Jul 09 03:59:56.764254 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.763799 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpnwv\" (UniqueName: \"kubernetes.io/projected/3c16341e-9824-4abb-af4d-1fa4d37acc1f-kube-api-access-rpnwv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2\" (UID: \"3c16341e-9824-4abb-af4d-1fa4d37acc1f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" Jul 09 03:59:56.764254 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.763925 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c16341e-9824-4abb-af4d-1fa4d37acc1f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2\" (UID: \"3c16341e-9824-4abb-af4d-1fa4d37acc1f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" Jul 09 03:59:56.865256 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.865224 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c16341e-9824-4abb-af4d-1fa4d37acc1f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2\" (UID: \"3c16341e-9824-4abb-af4d-1fa4d37acc1f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" Jul 09 03:59:56.865429 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.865262 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpnwv\" (UniqueName: \"kubernetes.io/projected/3c16341e-9824-4abb-af4d-1fa4d37acc1f-kube-api-access-rpnwv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2\" (UID: \"3c16341e-9824-4abb-af4d-1fa4d37acc1f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" Jul 09 03:59:56.865429 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.865307 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c16341e-9824-4abb-af4d-1fa4d37acc1f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2\" (UID: \"3c16341e-9824-4abb-af4d-1fa4d37acc1f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" Jul 09 03:59:56.865677 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.865662 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c16341e-9824-4abb-af4d-1fa4d37acc1f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2\" (UID: \"3c16341e-9824-4abb-af4d-1fa4d37acc1f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" Jul 09 03:59:56.865717 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.865675 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c16341e-9824-4abb-af4d-1fa4d37acc1f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2\" (UID: \"3c16341e-9824-4abb-af4d-1fa4d37acc1f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" Jul 09 03:59:56.873589 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.873516 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpnwv\" (UniqueName: \"kubernetes.io/projected/3c16341e-9824-4abb-af4d-1fa4d37acc1f-kube-api-access-rpnwv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2\" (UID: \"3c16341e-9824-4abb-af4d-1fa4d37acc1f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" Jul 09 03:59:56.930122 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.930091 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rv5vv"] Jul 09 03:59:56.933566 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.933549 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rv5vv" Jul 09 03:59:56.935674 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.935655 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-sfxcj\"" Jul 09 03:59:56.935770 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.935704 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 09 03:59:56.935770 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.935706 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 09 03:59:56.939143 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.939125 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" Jul 09 03:59:56.943209 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:56.943188 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rv5vv"] Jul 09 03:59:57.066141 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:57.066117 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2"] Jul 09 03:59:57.067330 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:57.067305 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab3f4eb3-4d81-44d6-ba0a-d87189d68a57-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rv5vv\" (UID: \"ab3f4eb3-4d81-44d6-ba0a-d87189d68a57\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rv5vv" Jul 09 03:59:57.067462 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:57.067363 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52frw\" (UniqueName: \"kubernetes.io/projected/ab3f4eb3-4d81-44d6-ba0a-d87189d68a57-kube-api-access-52frw\") pod \"cert-manager-webhook-6486ffd997-rv5vv\" (UID: \"ab3f4eb3-4d81-44d6-ba0a-d87189d68a57\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rv5vv" Jul 09 03:59:57.068527 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:59:57.068505 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c16341e_9824_4abb_af4d_1fa4d37acc1f.slice/crio-4abd880b02fe584943f1d15f06fcce898c2584840f65b71192195148d95899d2 WatchSource:0}: Error finding container 4abd880b02fe584943f1d15f06fcce898c2584840f65b71192195148d95899d2: Status 404 returned error can't find the container with id 4abd880b02fe584943f1d15f06fcce898c2584840f65b71192195148d95899d2 Jul 09 03:59:57.168038 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:57.168009 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab3f4eb3-4d81-44d6-ba0a-d87189d68a57-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rv5vv\" (UID: \"ab3f4eb3-4d81-44d6-ba0a-d87189d68a57\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rv5vv" Jul 09 03:59:57.168164 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:57.168086 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52frw\" (UniqueName: \"kubernetes.io/projected/ab3f4eb3-4d81-44d6-ba0a-d87189d68a57-kube-api-access-52frw\") pod \"cert-manager-webhook-6486ffd997-rv5vv\" (UID: \"ab3f4eb3-4d81-44d6-ba0a-d87189d68a57\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rv5vv" Jul 09 03:59:57.175703 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:57.175677 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab3f4eb3-4d81-44d6-ba0a-d87189d68a57-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rv5vv\" (UID: \"ab3f4eb3-4d81-44d6-ba0a-d87189d68a57\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rv5vv" Jul 09 03:59:57.175939 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:57.175921 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52frw\" (UniqueName: \"kubernetes.io/projected/ab3f4eb3-4d81-44d6-ba0a-d87189d68a57-kube-api-access-52frw\") pod \"cert-manager-webhook-6486ffd997-rv5vv\" (UID: \"ab3f4eb3-4d81-44d6-ba0a-d87189d68a57\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rv5vv" Jul 09 03:59:57.250059 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:57.250030 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rv5vv" Jul 09 03:59:57.374533 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:57.374377 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rv5vv"] Jul 09 03:59:57.377023 ip-10-0-130-55 kubenswrapper[2571]: W0709 03:59:57.376997 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab3f4eb3_4d81_44d6_ba0a_d87189d68a57.slice/crio-0168328a7d46dd8274b904967c2b8b58d8ea29c907f83209a04a777536f5d536 WatchSource:0}: Error finding container 0168328a7d46dd8274b904967c2b8b58d8ea29c907f83209a04a777536f5d536: Status 404 returned error can't find the container with id 0168328a7d46dd8274b904967c2b8b58d8ea29c907f83209a04a777536f5d536 Jul 09 03:59:57.720637 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:57.720599 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rv5vv" event={"ID":"ab3f4eb3-4d81-44d6-ba0a-d87189d68a57","Type":"ContainerStarted","Data":"0168328a7d46dd8274b904967c2b8b58d8ea29c907f83209a04a777536f5d536"} Jul 09 03:59:57.721963 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:57.721936 2571 generic.go:356] "Generic (PLEG): container finished" podID="3c16341e-9824-4abb-af4d-1fa4d37acc1f" containerID="6d6e34e26d8b4107eba83f6b1fa306ff91a0abd5cfd1a5970c17b03618bf383f" exitCode=0 Jul 09 03:59:57.722071 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:57.722003 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" event={"ID":"3c16341e-9824-4abb-af4d-1fa4d37acc1f","Type":"ContainerDied","Data":"6d6e34e26d8b4107eba83f6b1fa306ff91a0abd5cfd1a5970c17b03618bf383f"} Jul 09 03:59:57.722071 ip-10-0-130-55 kubenswrapper[2571]: I0709 03:59:57.722046 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" event={"ID":"3c16341e-9824-4abb-af4d-1fa4d37acc1f","Type":"ContainerStarted","Data":"4abd880b02fe584943f1d15f06fcce898c2584840f65b71192195148d95899d2"} Jul 09 04:00:01.745901 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:01.745845 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rv5vv" event={"ID":"ab3f4eb3-4d81-44d6-ba0a-d87189d68a57","Type":"ContainerStarted","Data":"6950737d2c1fbe2cc2324e2d22045df9d521f573e668577112bfe5362673990e"} Jul 09 04:00:01.746336 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:01.745939 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rv5vv" Jul 09 04:00:01.747662 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:01.747636 2571 generic.go:356] "Generic (PLEG): container finished" podID="3c16341e-9824-4abb-af4d-1fa4d37acc1f" containerID="b9181c30e47d8b726a1341c1b382d2af5bf4138ebfaf2b788a9a475e68b40a91" exitCode=0 Jul 09 04:00:01.747760 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:01.747714 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" event={"ID":"3c16341e-9824-4abb-af4d-1fa4d37acc1f","Type":"ContainerDied","Data":"b9181c30e47d8b726a1341c1b382d2af5bf4138ebfaf2b788a9a475e68b40a91"} Jul 09 04:00:01.763498 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:01.763450 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-rv5vv" podStartSLOduration=1.864801607 podStartE2EDuration="5.763431912s" totalImagesPullingTime="3.898630305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:59:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:01.763186059 +0000 UTC m=+236.622057160" watchObservedRunningTime="2026-07-09 04:00:01.763431912 +0000 UTC m=+236.622303013" Jul 09 04:00:02.754661 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:02.754633 2571 generic.go:356] "Generic (PLEG): container finished" podID="3c16341e-9824-4abb-af4d-1fa4d37acc1f" containerID="b373d6bba95edcbe9a31e3b4151c3f8bbcfe126d04b97c93a9431939ebe82be7" exitCode=0 Jul 09 04:00:02.755092 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:02.754760 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" event={"ID":"3c16341e-9824-4abb-af4d-1fa4d37acc1f","Type":"ContainerDied","Data":"b373d6bba95edcbe9a31e3b4151c3f8bbcfe126d04b97c93a9431939ebe82be7"} Jul 09 04:00:03.889674 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:03.889646 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" Jul 09 04:00:03.917658 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:03.917626 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2pdhb"] Jul 09 04:00:03.918179 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:03.918159 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c16341e-9824-4abb-af4d-1fa4d37acc1f" containerName="util" Jul 09 04:00:03.918240 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:03.918182 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c16341e-9824-4abb-af4d-1fa4d37acc1f" containerName="util" Jul 09 04:00:03.918240 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:03.918208 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c16341e-9824-4abb-af4d-1fa4d37acc1f" containerName="pull" Jul 09 04:00:03.918240 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:03.918217 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c16341e-9824-4abb-af4d-1fa4d37acc1f" containerName="pull" Jul 09 04:00:03.918240 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:03.918235 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c16341e-9824-4abb-af4d-1fa4d37acc1f" containerName="extract" Jul 09 04:00:03.918364 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:03.918243 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c16341e-9824-4abb-af4d-1fa4d37acc1f" containerName="extract" Jul 09 04:00:03.918364 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:03.918341 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c16341e-9824-4abb-af4d-1fa4d37acc1f" containerName="extract" Jul 09 04:00:03.920574 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:03.920556 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2pdhb" Jul 09 04:00:03.922994 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:03.922971 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-hk4dg\"" Jul 09 04:00:03.928224 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:03.928204 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2pdhb"] Jul 09 04:00:04.033211 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.033123 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpnwv\" (UniqueName: \"kubernetes.io/projected/3c16341e-9824-4abb-af4d-1fa4d37acc1f-kube-api-access-rpnwv\") pod \"3c16341e-9824-4abb-af4d-1fa4d37acc1f\" (UID: \"3c16341e-9824-4abb-af4d-1fa4d37acc1f\") " Jul 09 04:00:04.033211 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.033206 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c16341e-9824-4abb-af4d-1fa4d37acc1f-util\") pod \"3c16341e-9824-4abb-af4d-1fa4d37acc1f\" (UID: \"3c16341e-9824-4abb-af4d-1fa4d37acc1f\") " Jul 09 04:00:04.033398 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.033226 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c16341e-9824-4abb-af4d-1fa4d37acc1f-bundle\") pod \"3c16341e-9824-4abb-af4d-1fa4d37acc1f\" (UID: \"3c16341e-9824-4abb-af4d-1fa4d37acc1f\") " Jul 09 04:00:04.033398 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.033309 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkp5g\" (UniqueName: \"kubernetes.io/projected/8d670825-c2a2-4fa4-8a3c-28f2e16a2a44-kube-api-access-hkp5g\") pod \"cert-manager-cainjector-559cdd8c87-2pdhb\" (UID: \"8d670825-c2a2-4fa4-8a3c-28f2e16a2a44\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2pdhb" Jul 09 04:00:04.033398 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.033337 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d670825-c2a2-4fa4-8a3c-28f2e16a2a44-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2pdhb\" (UID: \"8d670825-c2a2-4fa4-8a3c-28f2e16a2a44\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2pdhb" Jul 09 04:00:04.033656 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.033625 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c16341e-9824-4abb-af4d-1fa4d37acc1f-bundle" (OuterVolumeSpecName: "bundle") pod "3c16341e-9824-4abb-af4d-1fa4d37acc1f" (UID: "3c16341e-9824-4abb-af4d-1fa4d37acc1f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:00:04.035323 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.035287 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c16341e-9824-4abb-af4d-1fa4d37acc1f-kube-api-access-rpnwv" (OuterVolumeSpecName: "kube-api-access-rpnwv") pod "3c16341e-9824-4abb-af4d-1fa4d37acc1f" (UID: "3c16341e-9824-4abb-af4d-1fa4d37acc1f"). InnerVolumeSpecName "kube-api-access-rpnwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:00:04.037878 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.037839 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c16341e-9824-4abb-af4d-1fa4d37acc1f-util" (OuterVolumeSpecName: "util") pod "3c16341e-9824-4abb-af4d-1fa4d37acc1f" (UID: "3c16341e-9824-4abb-af4d-1fa4d37acc1f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:00:04.133935 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.133889 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkp5g\" (UniqueName: \"kubernetes.io/projected/8d670825-c2a2-4fa4-8a3c-28f2e16a2a44-kube-api-access-hkp5g\") pod \"cert-manager-cainjector-559cdd8c87-2pdhb\" (UID: \"8d670825-c2a2-4fa4-8a3c-28f2e16a2a44\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2pdhb" Jul 09 04:00:04.133935 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.133934 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d670825-c2a2-4fa4-8a3c-28f2e16a2a44-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2pdhb\" (UID: \"8d670825-c2a2-4fa4-8a3c-28f2e16a2a44\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2pdhb" Jul 09 04:00:04.134148 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.134000 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpnwv\" (UniqueName: \"kubernetes.io/projected/3c16341e-9824-4abb-af4d-1fa4d37acc1f-kube-api-access-rpnwv\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:00:04.134148 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.134011 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c16341e-9824-4abb-af4d-1fa4d37acc1f-util\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:00:04.134148 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.134025 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c16341e-9824-4abb-af4d-1fa4d37acc1f-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:00:04.141665 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.141635 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d670825-c2a2-4fa4-8a3c-28f2e16a2a44-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2pdhb\" (UID: \"8d670825-c2a2-4fa4-8a3c-28f2e16a2a44\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2pdhb" Jul 09 04:00:04.141955 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.141934 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkp5g\" (UniqueName: \"kubernetes.io/projected/8d670825-c2a2-4fa4-8a3c-28f2e16a2a44-kube-api-access-hkp5g\") pod \"cert-manager-cainjector-559cdd8c87-2pdhb\" (UID: \"8d670825-c2a2-4fa4-8a3c-28f2e16a2a44\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2pdhb" Jul 09 04:00:04.230587 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.230547 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2pdhb" Jul 09 04:00:04.359514 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.359460 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2pdhb"] Jul 09 04:00:04.361609 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:00:04.361584 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d670825_c2a2_4fa4_8a3c_28f2e16a2a44.slice/crio-d82d447cadc6cf71f674d334c502f7b5d4c4441c1d8527631ac3a4154584094f WatchSource:0}: Error finding container d82d447cadc6cf71f674d334c502f7b5d4c4441c1d8527631ac3a4154584094f: Status 404 returned error can't find the container with id d82d447cadc6cf71f674d334c502f7b5d4c4441c1d8527631ac3a4154584094f Jul 09 04:00:04.767532 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.767501 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" Jul 09 04:00:04.767712 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.767503 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fchcj2" event={"ID":"3c16341e-9824-4abb-af4d-1fa4d37acc1f","Type":"ContainerDied","Data":"4abd880b02fe584943f1d15f06fcce898c2584840f65b71192195148d95899d2"} Jul 09 04:00:04.767712 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.767630 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4abd880b02fe584943f1d15f06fcce898c2584840f65b71192195148d95899d2" Jul 09 04:00:04.769246 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.769222 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2pdhb" event={"ID":"8d670825-c2a2-4fa4-8a3c-28f2e16a2a44","Type":"ContainerStarted","Data":"70af688c16b2a9c9e0b9000541fbca84848aaba04366badf37d6c12de793814a"} Jul 09 04:00:04.769340 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.769255 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2pdhb" event={"ID":"8d670825-c2a2-4fa4-8a3c-28f2e16a2a44","Type":"ContainerStarted","Data":"d82d447cadc6cf71f674d334c502f7b5d4c4441c1d8527631ac3a4154584094f"} Jul 09 04:00:04.785342 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:04.785270 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2pdhb" podStartSLOduration=1.785251784 podStartE2EDuration="1.785251784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:04.784501343 +0000 UTC m=+239.643372466" watchObservedRunningTime="2026-07-09 04:00:04.785251784 +0000 UTC m=+239.644122885" Jul 09 04:00:07.757460 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:07.757424 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rv5vv" Jul 09 04:00:22.469554 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.469519 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6"] Jul 09 04:00:22.474846 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.474826 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" Jul 09 04:00:22.476965 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.476943 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-d248r\"" Jul 09 04:00:22.477317 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.477262 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 04:00:22.478126 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.478099 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 04:00:22.480754 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.480734 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6"] Jul 09 04:00:22.492031 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.492003 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6\" (UID: \"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" Jul 09 04:00:22.492435 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.492406 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn6w5\" (UniqueName: \"kubernetes.io/projected/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-kube-api-access-wn6w5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6\" (UID: \"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" Jul 09 04:00:22.492549 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.492491 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6\" (UID: \"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" Jul 09 04:00:22.593137 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.593094 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6\" (UID: \"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" Jul 09 04:00:22.593303 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.593186 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6\" (UID: \"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" Jul 09 04:00:22.593303 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.593230 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn6w5\" (UniqueName: \"kubernetes.io/projected/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-kube-api-access-wn6w5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6\" (UID: \"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" Jul 09 04:00:22.593579 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.593555 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6\" (UID: \"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" Jul 09 04:00:22.593616 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.593562 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6\" (UID: \"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" Jul 09 04:00:22.601751 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.601717 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn6w5\" (UniqueName: \"kubernetes.io/projected/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-kube-api-access-wn6w5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6\" (UID: \"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" Jul 09 04:00:22.786279 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.786179 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" Jul 09 04:00:22.918810 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:22.918777 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6"] Jul 09 04:00:22.921774 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:00:22.921746 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1cd859_2d1d_49d1_b790_1e73e1bc60d4.slice/crio-62ad11ff6e40b7850fedef544d1bb6753844493b02bb43e1dd0bc76ade573a84 WatchSource:0}: Error finding container 62ad11ff6e40b7850fedef544d1bb6753844493b02bb43e1dd0bc76ade573a84: Status 404 returned error can't find the container with id 62ad11ff6e40b7850fedef544d1bb6753844493b02bb43e1dd0bc76ade573a84 Jul 09 04:00:23.859331 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:23.859247 2571 generic.go:356] "Generic (PLEG): container finished" podID="8d1cd859-2d1d-49d1-b790-1e73e1bc60d4" containerID="f7210ab7943c391b8d964b3f3231862bf8e82fa3d7f1cbc7786d1317f3005559" exitCode=0 Jul 09 04:00:23.859331 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:23.859316 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" event={"ID":"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4","Type":"ContainerDied","Data":"f7210ab7943c391b8d964b3f3231862bf8e82fa3d7f1cbc7786d1317f3005559"} Jul 09 04:00:23.859727 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:23.859357 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" event={"ID":"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4","Type":"ContainerStarted","Data":"62ad11ff6e40b7850fedef544d1bb6753844493b02bb43e1dd0bc76ade573a84"} Jul 09 04:00:24.865350 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:24.865316 2571 generic.go:356] "Generic (PLEG): container finished" podID="8d1cd859-2d1d-49d1-b790-1e73e1bc60d4" containerID="d61248b9d345a403211ba46b2326e598212261ba6939bbc72ed39b50d20056f0" exitCode=0 Jul 09 04:00:24.865701 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:24.865396 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" event={"ID":"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4","Type":"ContainerDied","Data":"d61248b9d345a403211ba46b2326e598212261ba6939bbc72ed39b50d20056f0"} Jul 09 04:00:25.871904 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:25.871867 2571 generic.go:356] "Generic (PLEG): container finished" podID="8d1cd859-2d1d-49d1-b790-1e73e1bc60d4" containerID="0065078ee2ccfc77b9862bc60f5c01035764287e9537dbe2af16f7f606b31188" exitCode=0 Jul 09 04:00:25.872280 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:25.871919 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" event={"ID":"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4","Type":"ContainerDied","Data":"0065078ee2ccfc77b9862bc60f5c01035764287e9537dbe2af16f7f606b31188"} Jul 09 04:00:27.010998 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:27.010972 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" Jul 09 04:00:27.025137 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:27.025109 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-util\") pod \"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4\" (UID: \"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4\") " Jul 09 04:00:27.025280 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:27.025159 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-bundle\") pod \"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4\" (UID: \"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4\") " Jul 09 04:00:27.025988 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:27.025944 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-bundle" (OuterVolumeSpecName: "bundle") pod "8d1cd859-2d1d-49d1-b790-1e73e1bc60d4" (UID: "8d1cd859-2d1d-49d1-b790-1e73e1bc60d4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:00:27.031788 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:27.031762 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-util" (OuterVolumeSpecName: "util") pod "8d1cd859-2d1d-49d1-b790-1e73e1bc60d4" (UID: "8d1cd859-2d1d-49d1-b790-1e73e1bc60d4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:00:27.125877 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:27.125767 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn6w5\" (UniqueName: \"kubernetes.io/projected/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-kube-api-access-wn6w5\") pod \"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4\" (UID: \"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4\") " Jul 09 04:00:27.126027 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:27.125941 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-util\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:00:27.126027 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:27.125952 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:00:27.127879 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:27.127793 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-kube-api-access-wn6w5" (OuterVolumeSpecName: "kube-api-access-wn6w5") pod "8d1cd859-2d1d-49d1-b790-1e73e1bc60d4" (UID: "8d1cd859-2d1d-49d1-b790-1e73e1bc60d4"). InnerVolumeSpecName "kube-api-access-wn6w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:00:27.232074 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:27.227796 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn6w5\" (UniqueName: \"kubernetes.io/projected/8d1cd859-2d1d-49d1-b790-1e73e1bc60d4-kube-api-access-wn6w5\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:00:27.883327 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:27.883236 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" event={"ID":"8d1cd859-2d1d-49d1-b790-1e73e1bc60d4","Type":"ContainerDied","Data":"62ad11ff6e40b7850fedef544d1bb6753844493b02bb43e1dd0bc76ade573a84"} Jul 09 04:00:27.883327 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:27.883284 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62ad11ff6e40b7850fedef544d1bb6753844493b02bb43e1dd0bc76ade573a84" Jul 09 04:00:27.883327 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:27.883297 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz25q6" Jul 09 04:00:34.548533 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.548488 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz"] Jul 09 04:00:34.549131 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.549001 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1cd859-2d1d-49d1-b790-1e73e1bc60d4" containerName="util" Jul 09 04:00:34.549131 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.549017 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1cd859-2d1d-49d1-b790-1e73e1bc60d4" containerName="util" Jul 09 04:00:34.549131 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.549029 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1cd859-2d1d-49d1-b790-1e73e1bc60d4" containerName="extract" Jul 09 04:00:34.549131 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.549038 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1cd859-2d1d-49d1-b790-1e73e1bc60d4" containerName="extract" Jul 09 04:00:34.549131 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.549061 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1cd859-2d1d-49d1-b790-1e73e1bc60d4" containerName="pull" Jul 09 04:00:34.549131 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.549066 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1cd859-2d1d-49d1-b790-1e73e1bc60d4" containerName="pull" Jul 09 04:00:34.549131 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.549133 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d1cd859-2d1d-49d1-b790-1e73e1bc60d4" containerName="extract" Jul 09 04:00:34.551768 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.551752 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:34.554134 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.554112 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 04:00:34.554277 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.554145 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mnmqz\"" Jul 09 04:00:34.554333 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.554284 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 04:00:34.554399 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.554380 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 04:00:34.554485 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.554417 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 04:00:34.567724 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.567690 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz"] Jul 09 04:00:34.584163 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.584131 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wcdp\" (UniqueName: \"kubernetes.io/projected/1dc27941-2ef5-42f7-95cc-8915108a811b-kube-api-access-8wcdp\") pod \"opendatahub-operator-controller-manager-7d756d6f6f-dhdjz\" (UID: \"1dc27941-2ef5-42f7-95cc-8915108a811b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:34.584344 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.584175 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1dc27941-2ef5-42f7-95cc-8915108a811b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d756d6f6f-dhdjz\" (UID: \"1dc27941-2ef5-42f7-95cc-8915108a811b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:34.584344 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.584296 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1dc27941-2ef5-42f7-95cc-8915108a811b-manifests\") pod \"opendatahub-operator-controller-manager-7d756d6f6f-dhdjz\" (UID: \"1dc27941-2ef5-42f7-95cc-8915108a811b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:34.584344 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.584331 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1dc27941-2ef5-42f7-95cc-8915108a811b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d756d6f6f-dhdjz\" (UID: \"1dc27941-2ef5-42f7-95cc-8915108a811b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:34.584469 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.584371 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1dc27941-2ef5-42f7-95cc-8915108a811b-tmp\") pod \"opendatahub-operator-controller-manager-7d756d6f6f-dhdjz\" (UID: \"1dc27941-2ef5-42f7-95cc-8915108a811b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:34.685431 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.685388 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1dc27941-2ef5-42f7-95cc-8915108a811b-manifests\") pod \"opendatahub-operator-controller-manager-7d756d6f6f-dhdjz\" (UID: \"1dc27941-2ef5-42f7-95cc-8915108a811b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:34.685431 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.685432 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1dc27941-2ef5-42f7-95cc-8915108a811b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d756d6f6f-dhdjz\" (UID: \"1dc27941-2ef5-42f7-95cc-8915108a811b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:34.685702 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.685484 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1dc27941-2ef5-42f7-95cc-8915108a811b-tmp\") pod \"opendatahub-operator-controller-manager-7d756d6f6f-dhdjz\" (UID: \"1dc27941-2ef5-42f7-95cc-8915108a811b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:34.685702 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.685524 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wcdp\" (UniqueName: \"kubernetes.io/projected/1dc27941-2ef5-42f7-95cc-8915108a811b-kube-api-access-8wcdp\") pod \"opendatahub-operator-controller-manager-7d756d6f6f-dhdjz\" (UID: \"1dc27941-2ef5-42f7-95cc-8915108a811b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:34.685702 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.685553 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1dc27941-2ef5-42f7-95cc-8915108a811b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d756d6f6f-dhdjz\" (UID: \"1dc27941-2ef5-42f7-95cc-8915108a811b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:34.685911 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.685845 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1dc27941-2ef5-42f7-95cc-8915108a811b-manifests\") pod \"opendatahub-operator-controller-manager-7d756d6f6f-dhdjz\" (UID: \"1dc27941-2ef5-42f7-95cc-8915108a811b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:34.687817 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.687790 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1dc27941-2ef5-42f7-95cc-8915108a811b-tmp\") pod \"opendatahub-operator-controller-manager-7d756d6f6f-dhdjz\" (UID: \"1dc27941-2ef5-42f7-95cc-8915108a811b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:34.687964 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.687948 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1dc27941-2ef5-42f7-95cc-8915108a811b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d756d6f6f-dhdjz\" (UID: \"1dc27941-2ef5-42f7-95cc-8915108a811b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:34.688139 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.688116 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1dc27941-2ef5-42f7-95cc-8915108a811b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d756d6f6f-dhdjz\" (UID: \"1dc27941-2ef5-42f7-95cc-8915108a811b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:34.695961 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.695936 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wcdp\" (UniqueName: \"kubernetes.io/projected/1dc27941-2ef5-42f7-95cc-8915108a811b-kube-api-access-8wcdp\") pod \"opendatahub-operator-controller-manager-7d756d6f6f-dhdjz\" (UID: \"1dc27941-2ef5-42f7-95cc-8915108a811b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:34.862010 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:34.861972 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:35.011634 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:35.011607 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz"] Jul 09 04:00:35.014195 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:00:35.014170 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc27941_2ef5_42f7_95cc_8915108a811b.slice/crio-aee56478a0242507f77ad408754ed7fabec44aa28846d725b6a3d267aaf9ec9c WatchSource:0}: Error finding container aee56478a0242507f77ad408754ed7fabec44aa28846d725b6a3d267aaf9ec9c: Status 404 returned error can't find the container with id aee56478a0242507f77ad408754ed7fabec44aa28846d725b6a3d267aaf9ec9c Jul 09 04:00:35.922399 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:35.922366 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" event={"ID":"1dc27941-2ef5-42f7-95cc-8915108a811b","Type":"ContainerStarted","Data":"aee56478a0242507f77ad408754ed7fabec44aa28846d725b6a3d267aaf9ec9c"} Jul 09 04:00:38.940474 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:38.940443 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" event={"ID":"1dc27941-2ef5-42f7-95cc-8915108a811b","Type":"ContainerStarted","Data":"afb4f0cf5c6f5fad953ab70a829736a0ad8877a3b6be921d122f77a054d39edc"} Jul 09 04:00:39.946408 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:39.946373 2571 generic.go:356] "Generic (PLEG): container finished" podID="1dc27941-2ef5-42f7-95cc-8915108a811b" containerID="afb4f0cf5c6f5fad953ab70a829736a0ad8877a3b6be921d122f77a054d39edc" exitCode=0 Jul 09 04:00:39.946789 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:39.946458 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" event={"ID":"1dc27941-2ef5-42f7-95cc-8915108a811b","Type":"ContainerDied","Data":"afb4f0cf5c6f5fad953ab70a829736a0ad8877a3b6be921d122f77a054d39edc"} Jul 09 04:00:40.955876 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:40.955820 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" event={"ID":"1dc27941-2ef5-42f7-95cc-8915108a811b","Type":"ContainerStarted","Data":"7b53a7fe723372d862426756e3eb7190bd05e9670115b6a09c7006e7cfc9ce7b"} Jul 09 04:00:40.956283 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:40.955951 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:41.006695 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:41.006653 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" podStartSLOduration=2.210106354 podStartE2EDuration="7.006638356s" totalImagesPullingTime="3.709648381s" totalInitContainerRuntime="1.086883621s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:41.006025429 +0000 UTC m=+275.864896539" watchObservedRunningTime="2026-07-09 04:00:41.006638356 +0000 UTC m=+275.865509455" Jul 09 04:00:51.963937 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:51.963910 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d756d6f6f-dhdjz" Jul 09 04:00:54.524621 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:54.524588 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-644c76db6f-vttfs"] Jul 09 04:00:54.633609 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:54.633573 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-644c76db6f-vttfs"] Jul 09 04:00:54.633753 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:54.633686 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-644c76db6f-vttfs" Jul 09 04:00:54.636693 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:54.636669 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4z6hc\"" Jul 09 04:00:54.636693 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:54.636680 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 09 04:00:54.636905 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:54.636669 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 04:00:54.767989 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:54.767957 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwth6\" (UniqueName: \"kubernetes.io/projected/d4b22e9d-d1ce-4539-aff3-e14c37347ba1-kube-api-access-mwth6\") pod \"kube-auth-proxy-644c76db6f-vttfs\" (UID: \"d4b22e9d-d1ce-4539-aff3-e14c37347ba1\") " pod="openshift-ingress/kube-auth-proxy-644c76db6f-vttfs" Jul 09 04:00:54.768156 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:54.767997 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4b22e9d-d1ce-4539-aff3-e14c37347ba1-tmp\") pod \"kube-auth-proxy-644c76db6f-vttfs\" (UID: \"d4b22e9d-d1ce-4539-aff3-e14c37347ba1\") " pod="openshift-ingress/kube-auth-proxy-644c76db6f-vttfs" Jul 09 04:00:54.768156 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:54.768078 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b22e9d-d1ce-4539-aff3-e14c37347ba1-tls-certs\") pod \"kube-auth-proxy-644c76db6f-vttfs\" (UID: \"d4b22e9d-d1ce-4539-aff3-e14c37347ba1\") " pod="openshift-ingress/kube-auth-proxy-644c76db6f-vttfs" Jul 09 04:00:54.868575 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:54.868528 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwth6\" (UniqueName: \"kubernetes.io/projected/d4b22e9d-d1ce-4539-aff3-e14c37347ba1-kube-api-access-mwth6\") pod \"kube-auth-proxy-644c76db6f-vttfs\" (UID: \"d4b22e9d-d1ce-4539-aff3-e14c37347ba1\") " pod="openshift-ingress/kube-auth-proxy-644c76db6f-vttfs" Jul 09 04:00:54.868800 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:54.868593 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4b22e9d-d1ce-4539-aff3-e14c37347ba1-tmp\") pod \"kube-auth-proxy-644c76db6f-vttfs\" (UID: \"d4b22e9d-d1ce-4539-aff3-e14c37347ba1\") " pod="openshift-ingress/kube-auth-proxy-644c76db6f-vttfs" Jul 09 04:00:54.868800 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:54.868657 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b22e9d-d1ce-4539-aff3-e14c37347ba1-tls-certs\") pod \"kube-auth-proxy-644c76db6f-vttfs\" (UID: \"d4b22e9d-d1ce-4539-aff3-e14c37347ba1\") " pod="openshift-ingress/kube-auth-proxy-644c76db6f-vttfs" Jul 09 04:00:54.871497 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:54.871450 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4b22e9d-d1ce-4539-aff3-e14c37347ba1-tmp\") pod \"kube-auth-proxy-644c76db6f-vttfs\" (UID: \"d4b22e9d-d1ce-4539-aff3-e14c37347ba1\") " pod="openshift-ingress/kube-auth-proxy-644c76db6f-vttfs" Jul 09 04:00:54.871631 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:54.871610 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b22e9d-d1ce-4539-aff3-e14c37347ba1-tls-certs\") pod \"kube-auth-proxy-644c76db6f-vttfs\" (UID: \"d4b22e9d-d1ce-4539-aff3-e14c37347ba1\") " pod="openshift-ingress/kube-auth-proxy-644c76db6f-vttfs" Jul 09 04:00:54.880049 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:54.880025 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwth6\" (UniqueName: \"kubernetes.io/projected/d4b22e9d-d1ce-4539-aff3-e14c37347ba1-kube-api-access-mwth6\") pod \"kube-auth-proxy-644c76db6f-vttfs\" (UID: \"d4b22e9d-d1ce-4539-aff3-e14c37347ba1\") " pod="openshift-ingress/kube-auth-proxy-644c76db6f-vttfs" Jul 09 04:00:54.943870 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:54.943831 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-644c76db6f-vttfs" Jul 09 04:00:55.101186 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.101147 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-644c76db6f-vttfs"] Jul 09 04:00:55.103343 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:00:55.103316 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b22e9d_d1ce_4539_aff3_e14c37347ba1.slice/crio-cea10355e8b6b75a39ef24662dc64da35cd7f298bea8196a8f202f7e3ea6438b WatchSource:0}: Error finding container cea10355e8b6b75a39ef24662dc64da35cd7f298bea8196a8f202f7e3ea6438b: Status 404 returned error can't find the container with id cea10355e8b6b75a39ef24662dc64da35cd7f298bea8196a8f202f7e3ea6438b Jul 09 04:00:55.360347 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.360311 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f"] Jul 09 04:00:55.363737 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.363710 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" Jul 09 04:00:55.366900 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.366691 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 09 04:00:55.366900 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.366732 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 09 04:00:55.366900 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.366764 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-csmxc\"" Jul 09 04:00:55.366900 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.366772 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 09 04:00:55.367200 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.367000 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 04:00:55.367200 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.367057 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 04:00:55.383163 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.383139 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f"] Jul 09 04:00:55.474824 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.474790 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4jck\" (UniqueName: \"kubernetes.io/projected/d647fa53-001a-40c1-8440-fe68a50e9903-kube-api-access-d4jck\") pod \"lws-controller-manager-789458d7cb-x5n6f\" (UID: \"d647fa53-001a-40c1-8440-fe68a50e9903\") " pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" Jul 09 04:00:55.475056 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.474828 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d647fa53-001a-40c1-8440-fe68a50e9903-manager-config\") pod \"lws-controller-manager-789458d7cb-x5n6f\" (UID: \"d647fa53-001a-40c1-8440-fe68a50e9903\") " pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" Jul 09 04:00:55.475056 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.474892 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d647fa53-001a-40c1-8440-fe68a50e9903-cert\") pod \"lws-controller-manager-789458d7cb-x5n6f\" (UID: \"d647fa53-001a-40c1-8440-fe68a50e9903\") " pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" Jul 09 04:00:55.475056 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.474998 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d647fa53-001a-40c1-8440-fe68a50e9903-metrics-cert\") pod \"lws-controller-manager-789458d7cb-x5n6f\" (UID: \"d647fa53-001a-40c1-8440-fe68a50e9903\") " pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" Jul 09 04:00:55.576071 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.576043 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d647fa53-001a-40c1-8440-fe68a50e9903-metrics-cert\") pod \"lws-controller-manager-789458d7cb-x5n6f\" (UID: \"d647fa53-001a-40c1-8440-fe68a50e9903\") " pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" Jul 09 04:00:55.576480 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.576161 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4jck\" (UniqueName: \"kubernetes.io/projected/d647fa53-001a-40c1-8440-fe68a50e9903-kube-api-access-d4jck\") pod \"lws-controller-manager-789458d7cb-x5n6f\" (UID: \"d647fa53-001a-40c1-8440-fe68a50e9903\") " pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" Jul 09 04:00:55.576480 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.576191 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d647fa53-001a-40c1-8440-fe68a50e9903-manager-config\") pod \"lws-controller-manager-789458d7cb-x5n6f\" (UID: \"d647fa53-001a-40c1-8440-fe68a50e9903\") " pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" Jul 09 04:00:55.576480 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.576212 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d647fa53-001a-40c1-8440-fe68a50e9903-cert\") pod \"lws-controller-manager-789458d7cb-x5n6f\" (UID: \"d647fa53-001a-40c1-8440-fe68a50e9903\") " pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" Jul 09 04:00:55.576869 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.576836 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d647fa53-001a-40c1-8440-fe68a50e9903-manager-config\") pod \"lws-controller-manager-789458d7cb-x5n6f\" (UID: \"d647fa53-001a-40c1-8440-fe68a50e9903\") " pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" Jul 09 04:00:55.578658 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.578639 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d647fa53-001a-40c1-8440-fe68a50e9903-metrics-cert\") pod \"lws-controller-manager-789458d7cb-x5n6f\" (UID: \"d647fa53-001a-40c1-8440-fe68a50e9903\") " pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" Jul 09 04:00:55.578738 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.578639 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d647fa53-001a-40c1-8440-fe68a50e9903-cert\") pod \"lws-controller-manager-789458d7cb-x5n6f\" (UID: \"d647fa53-001a-40c1-8440-fe68a50e9903\") " pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" Jul 09 04:00:55.606055 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.606025 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4jck\" (UniqueName: \"kubernetes.io/projected/d647fa53-001a-40c1-8440-fe68a50e9903-kube-api-access-d4jck\") pod \"lws-controller-manager-789458d7cb-x5n6f\" (UID: \"d647fa53-001a-40c1-8440-fe68a50e9903\") " pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" Jul 09 04:00:55.674734 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.674657 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" Jul 09 04:00:55.820142 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:55.820107 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f"] Jul 09 04:00:55.823355 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:00:55.823331 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd647fa53_001a_40c1_8440_fe68a50e9903.slice/crio-edf27a5f8d0ee8d435522d11ac92c861ed6e00cffbdbf7c29ddeffd88b183881 WatchSource:0}: Error finding container edf27a5f8d0ee8d435522d11ac92c861ed6e00cffbdbf7c29ddeffd88b183881: Status 404 returned error can't find the container with id edf27a5f8d0ee8d435522d11ac92c861ed6e00cffbdbf7c29ddeffd88b183881 Jul 09 04:00:56.029977 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:56.029888 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" event={"ID":"d647fa53-001a-40c1-8440-fe68a50e9903","Type":"ContainerStarted","Data":"edf27a5f8d0ee8d435522d11ac92c861ed6e00cffbdbf7c29ddeffd88b183881"} Jul 09 04:00:56.031220 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:56.031194 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-644c76db6f-vttfs" event={"ID":"d4b22e9d-d1ce-4539-aff3-e14c37347ba1","Type":"ContainerStarted","Data":"cea10355e8b6b75a39ef24662dc64da35cd7f298bea8196a8f202f7e3ea6438b"} Jul 09 04:00:59.050805 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:59.050769 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-644c76db6f-vttfs" event={"ID":"d4b22e9d-d1ce-4539-aff3-e14c37347ba1","Type":"ContainerStarted","Data":"aef9fc8d04541f8224ff3733e0843e91d6c5ab3f1d5af74a8cc57f9e6275b830"} Jul 09 04:00:59.066678 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:00:59.066633 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-644c76db6f-vttfs" podStartSLOduration=1.993314307 podStartE2EDuration="5.066616646s" totalImagesPullingTime="3.073302339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:59.064663775 +0000 UTC m=+293.923534878" watchObservedRunningTime="2026-07-09 04:00:59.066616646 +0000 UTC m=+293.925487746" Jul 09 04:01:03.072864 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:01:03.072814 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" event={"ID":"d647fa53-001a-40c1-8440-fe68a50e9903","Type":"ContainerStarted","Data":"4b41863b47f3e80b02dcb7f7317aedc12e0c57e8ea8cf780ba335018f0c71c33"} Jul 09 04:01:03.073253 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:01:03.072945 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" Jul 09 04:01:03.091940 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:01:03.091843 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" podStartSLOduration=0.943719107 podStartE2EDuration="8.091828306s" totalImagesPullingTime="7.148109199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:03.090790426 +0000 UTC m=+297.949661527" watchObservedRunningTime="2026-07-09 04:01:03.091828306 +0000 UTC m=+297.950699406" Jul 09 04:01:05.518755 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:01:05.518727 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:01:05.525461 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:01:05.525421 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:01:05.530053 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:01:05.530033 2571 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 04:01:14.081594 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:01:14.081565 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-789458d7cb-x5n6f" Jul 09 04:02:10.650879 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:10.650779 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx"] Jul 09 04:02:10.654208 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:10.654192 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" Jul 09 04:02:10.656408 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:10.656385 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 04:02:10.656408 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:10.656398 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 04:02:10.657063 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:10.657048 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-d4mf4\"" Jul 09 04:02:10.665031 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:10.665008 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx"] Jul 09 04:02:10.724762 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:10.724726 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2924267d-1e47-42d4-8651-6fc55a0aaa22-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx\" (UID: \"2924267d-1e47-42d4-8651-6fc55a0aaa22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" Jul 09 04:02:10.724762 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:10.724768 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2924267d-1e47-42d4-8651-6fc55a0aaa22-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx\" (UID: \"2924267d-1e47-42d4-8651-6fc55a0aaa22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" Jul 09 04:02:10.725005 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:10.724791 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84lzt\" (UniqueName: \"kubernetes.io/projected/2924267d-1e47-42d4-8651-6fc55a0aaa22-kube-api-access-84lzt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx\" (UID: \"2924267d-1e47-42d4-8651-6fc55a0aaa22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" Jul 09 04:02:10.826066 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:10.826033 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2924267d-1e47-42d4-8651-6fc55a0aaa22-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx\" (UID: \"2924267d-1e47-42d4-8651-6fc55a0aaa22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" Jul 09 04:02:10.826254 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:10.826073 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2924267d-1e47-42d4-8651-6fc55a0aaa22-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx\" (UID: \"2924267d-1e47-42d4-8651-6fc55a0aaa22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" Jul 09 04:02:10.826254 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:10.826096 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84lzt\" (UniqueName: \"kubernetes.io/projected/2924267d-1e47-42d4-8651-6fc55a0aaa22-kube-api-access-84lzt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx\" (UID: \"2924267d-1e47-42d4-8651-6fc55a0aaa22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" Jul 09 04:02:10.826452 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:10.826429 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2924267d-1e47-42d4-8651-6fc55a0aaa22-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx\" (UID: \"2924267d-1e47-42d4-8651-6fc55a0aaa22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" Jul 09 04:02:10.826553 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:10.826533 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2924267d-1e47-42d4-8651-6fc55a0aaa22-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx\" (UID: \"2924267d-1e47-42d4-8651-6fc55a0aaa22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" Jul 09 04:02:10.834162 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:10.834140 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84lzt\" (UniqueName: \"kubernetes.io/projected/2924267d-1e47-42d4-8651-6fc55a0aaa22-kube-api-access-84lzt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx\" (UID: \"2924267d-1e47-42d4-8651-6fc55a0aaa22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" Jul 09 04:02:10.963971 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:10.963845 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" Jul 09 04:02:11.055242 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.055204 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4"] Jul 09 04:02:11.059646 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.059627 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" Jul 09 04:02:11.065476 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.065441 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4"] Jul 09 04:02:11.098266 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.098235 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx"] Jul 09 04:02:11.101954 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:02:11.101920 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2924267d_1e47_42d4_8651_6fc55a0aaa22.slice/crio-dcfcb369aa2e84c066ec35bcb08195d8f8d7a11ff0f98b7f9c9b8b406e9ed6f2 WatchSource:0}: Error finding container dcfcb369aa2e84c066ec35bcb08195d8f8d7a11ff0f98b7f9c9b8b406e9ed6f2: Status 404 returned error can't find the container with id dcfcb369aa2e84c066ec35bcb08195d8f8d7a11ff0f98b7f9c9b8b406e9ed6f2 Jul 09 04:02:11.128233 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.128210 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82053c64-4844-42ae-83e3-fab214a4f537-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4\" (UID: \"82053c64-4844-42ae-83e3-fab214a4f537\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" Jul 09 04:02:11.128335 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.128246 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82053c64-4844-42ae-83e3-fab214a4f537-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4\" (UID: \"82053c64-4844-42ae-83e3-fab214a4f537\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" Jul 09 04:02:11.128335 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.128290 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rkp6\" (UniqueName: \"kubernetes.io/projected/82053c64-4844-42ae-83e3-fab214a4f537-kube-api-access-6rkp6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4\" (UID: \"82053c64-4844-42ae-83e3-fab214a4f537\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" Jul 09 04:02:11.228811 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.228739 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rkp6\" (UniqueName: \"kubernetes.io/projected/82053c64-4844-42ae-83e3-fab214a4f537-kube-api-access-6rkp6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4\" (UID: \"82053c64-4844-42ae-83e3-fab214a4f537\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" Jul 09 04:02:11.228966 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.228808 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82053c64-4844-42ae-83e3-fab214a4f537-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4\" (UID: \"82053c64-4844-42ae-83e3-fab214a4f537\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" Jul 09 04:02:11.228966 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.228865 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82053c64-4844-42ae-83e3-fab214a4f537-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4\" (UID: \"82053c64-4844-42ae-83e3-fab214a4f537\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" Jul 09 04:02:11.229172 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.229153 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82053c64-4844-42ae-83e3-fab214a4f537-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4\" (UID: \"82053c64-4844-42ae-83e3-fab214a4f537\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" Jul 09 04:02:11.229213 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.229190 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82053c64-4844-42ae-83e3-fab214a4f537-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4\" (UID: \"82053c64-4844-42ae-83e3-fab214a4f537\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" Jul 09 04:02:11.237946 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.237929 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rkp6\" (UniqueName: \"kubernetes.io/projected/82053c64-4844-42ae-83e3-fab214a4f537-kube-api-access-6rkp6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4\" (UID: \"82053c64-4844-42ae-83e3-fab214a4f537\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" Jul 09 04:02:11.372604 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.372558 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" Jul 09 04:02:11.397723 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.397688 2571 generic.go:356] "Generic (PLEG): container finished" podID="2924267d-1e47-42d4-8651-6fc55a0aaa22" containerID="b68a1fd108c4552a7de59ce2e39ed87a85b2aca610a21b4751edca882fb38468" exitCode=0 Jul 09 04:02:11.397876 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.397773 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" event={"ID":"2924267d-1e47-42d4-8651-6fc55a0aaa22","Type":"ContainerDied","Data":"b68a1fd108c4552a7de59ce2e39ed87a85b2aca610a21b4751edca882fb38468"} Jul 09 04:02:11.397876 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.397836 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" event={"ID":"2924267d-1e47-42d4-8651-6fc55a0aaa22","Type":"ContainerStarted","Data":"dcfcb369aa2e84c066ec35bcb08195d8f8d7a11ff0f98b7f9c9b8b406e9ed6f2"} Jul 09 04:02:11.398304 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.398289 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 04:02:11.504241 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.504217 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4"] Jul 09 04:02:11.506160 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:02:11.506138 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82053c64_4844_42ae_83e3_fab214a4f537.slice/crio-ce8c9a2a4cd56da00206531a36cbe0ca4adbbd2b5abd6ebca30841ac04b428e9 WatchSource:0}: Error finding container ce8c9a2a4cd56da00206531a36cbe0ca4adbbd2b5abd6ebca30841ac04b428e9: Status 404 returned error can't find the container with id ce8c9a2a4cd56da00206531a36cbe0ca4adbbd2b5abd6ebca30841ac04b428e9 Jul 09 04:02:11.649267 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.649238 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn"] Jul 09 04:02:11.652627 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.652609 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" Jul 09 04:02:11.660806 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.660781 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn"] Jul 09 04:02:11.734736 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.734699 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7crx\" (UniqueName: \"kubernetes.io/projected/77ed061d-f09c-4e24-8b76-ed05fef5630f-kube-api-access-m7crx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn\" (UID: \"77ed061d-f09c-4e24-8b76-ed05fef5630f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" Jul 09 04:02:11.734931 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.734769 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77ed061d-f09c-4e24-8b76-ed05fef5630f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn\" (UID: \"77ed061d-f09c-4e24-8b76-ed05fef5630f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" Jul 09 04:02:11.734931 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.734801 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77ed061d-f09c-4e24-8b76-ed05fef5630f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn\" (UID: \"77ed061d-f09c-4e24-8b76-ed05fef5630f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" Jul 09 04:02:11.835608 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.835536 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77ed061d-f09c-4e24-8b76-ed05fef5630f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn\" (UID: \"77ed061d-f09c-4e24-8b76-ed05fef5630f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" Jul 09 04:02:11.835608 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.835579 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77ed061d-f09c-4e24-8b76-ed05fef5630f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn\" (UID: \"77ed061d-f09c-4e24-8b76-ed05fef5630f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" Jul 09 04:02:11.835783 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.835634 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7crx\" (UniqueName: \"kubernetes.io/projected/77ed061d-f09c-4e24-8b76-ed05fef5630f-kube-api-access-m7crx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn\" (UID: \"77ed061d-f09c-4e24-8b76-ed05fef5630f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" Jul 09 04:02:11.835920 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.835903 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77ed061d-f09c-4e24-8b76-ed05fef5630f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn\" (UID: \"77ed061d-f09c-4e24-8b76-ed05fef5630f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" Jul 09 04:02:11.836025 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.836006 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77ed061d-f09c-4e24-8b76-ed05fef5630f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn\" (UID: \"77ed061d-f09c-4e24-8b76-ed05fef5630f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" Jul 09 04:02:11.844361 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.844336 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7crx\" (UniqueName: \"kubernetes.io/projected/77ed061d-f09c-4e24-8b76-ed05fef5630f-kube-api-access-m7crx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn\" (UID: \"77ed061d-f09c-4e24-8b76-ed05fef5630f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" Jul 09 04:02:11.962010 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:11.961982 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" Jul 09 04:02:12.092446 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.092421 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn"] Jul 09 04:02:12.093829 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:02:12.093799 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ed061d_f09c_4e24_8b76_ed05fef5630f.slice/crio-b51e4e1886d70834e3637e7e384b8284833b1ef83bb39cfb933baf11be5b9964 WatchSource:0}: Error finding container b51e4e1886d70834e3637e7e384b8284833b1ef83bb39cfb933baf11be5b9964: Status 404 returned error can't find the container with id b51e4e1886d70834e3637e7e384b8284833b1ef83bb39cfb933baf11be5b9964 Jul 09 04:02:12.251807 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.251772 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2"] Jul 09 04:02:12.255328 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.255311 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" Jul 09 04:02:12.261983 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.261962 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2"] Jul 09 04:02:12.340371 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.340332 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnd9j\" (UniqueName: \"kubernetes.io/projected/85f28e4c-7cbb-4214-9095-e10b146b8c95-kube-api-access-hnd9j\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2\" (UID: \"85f28e4c-7cbb-4214-9095-e10b146b8c95\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" Jul 09 04:02:12.340544 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.340439 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85f28e4c-7cbb-4214-9095-e10b146b8c95-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2\" (UID: \"85f28e4c-7cbb-4214-9095-e10b146b8c95\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" Jul 09 04:02:12.340544 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.340484 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85f28e4c-7cbb-4214-9095-e10b146b8c95-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2\" (UID: \"85f28e4c-7cbb-4214-9095-e10b146b8c95\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" Jul 09 04:02:12.403906 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.403790 2571 generic.go:356] "Generic (PLEG): container finished" podID="77ed061d-f09c-4e24-8b76-ed05fef5630f" containerID="c08fbd0d324049af69d008b58e7bf8f11a4b9cc7658409a79985d763538dd4b4" exitCode=0 Jul 09 04:02:12.404082 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.403896 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" event={"ID":"77ed061d-f09c-4e24-8b76-ed05fef5630f","Type":"ContainerDied","Data":"c08fbd0d324049af69d008b58e7bf8f11a4b9cc7658409a79985d763538dd4b4"} Jul 09 04:02:12.404082 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.403950 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" event={"ID":"77ed061d-f09c-4e24-8b76-ed05fef5630f","Type":"ContainerStarted","Data":"b51e4e1886d70834e3637e7e384b8284833b1ef83bb39cfb933baf11be5b9964"} Jul 09 04:02:12.405460 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.405434 2571 generic.go:356] "Generic (PLEG): container finished" podID="82053c64-4844-42ae-83e3-fab214a4f537" containerID="e36a8d6ab58596054a519902197e43d9bc7995949c62ac8f03aa7afeff3a4c3c" exitCode=0 Jul 09 04:02:12.405561 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.405476 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" event={"ID":"82053c64-4844-42ae-83e3-fab214a4f537","Type":"ContainerDied","Data":"e36a8d6ab58596054a519902197e43d9bc7995949c62ac8f03aa7afeff3a4c3c"} Jul 09 04:02:12.405561 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.405510 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" event={"ID":"82053c64-4844-42ae-83e3-fab214a4f537","Type":"ContainerStarted","Data":"ce8c9a2a4cd56da00206531a36cbe0ca4adbbd2b5abd6ebca30841ac04b428e9"} Jul 09 04:02:12.441623 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.441594 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85f28e4c-7cbb-4214-9095-e10b146b8c95-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2\" (UID: \"85f28e4c-7cbb-4214-9095-e10b146b8c95\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" Jul 09 04:02:12.441771 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.441657 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnd9j\" (UniqueName: \"kubernetes.io/projected/85f28e4c-7cbb-4214-9095-e10b146b8c95-kube-api-access-hnd9j\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2\" (UID: \"85f28e4c-7cbb-4214-9095-e10b146b8c95\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" Jul 09 04:02:12.441771 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.441720 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85f28e4c-7cbb-4214-9095-e10b146b8c95-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2\" (UID: \"85f28e4c-7cbb-4214-9095-e10b146b8c95\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" Jul 09 04:02:12.442075 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.442042 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85f28e4c-7cbb-4214-9095-e10b146b8c95-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2\" (UID: \"85f28e4c-7cbb-4214-9095-e10b146b8c95\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" Jul 09 04:02:12.442155 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.442087 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85f28e4c-7cbb-4214-9095-e10b146b8c95-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2\" (UID: \"85f28e4c-7cbb-4214-9095-e10b146b8c95\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" Jul 09 04:02:12.450122 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.450103 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnd9j\" (UniqueName: \"kubernetes.io/projected/85f28e4c-7cbb-4214-9095-e10b146b8c95-kube-api-access-hnd9j\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2\" (UID: \"85f28e4c-7cbb-4214-9095-e10b146b8c95\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" Jul 09 04:02:12.566314 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.566286 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" Jul 09 04:02:12.809203 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:12.809155 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2"] Jul 09 04:02:12.811687 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:02:12.811656 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f28e4c_7cbb_4214_9095_e10b146b8c95.slice/crio-30fe964659229a3a3d9d74614c2334bedf5441d41f3755326b9a3f62522eefb6 WatchSource:0}: Error finding container 30fe964659229a3a3d9d74614c2334bedf5441d41f3755326b9a3f62522eefb6: Status 404 returned error can't find the container with id 30fe964659229a3a3d9d74614c2334bedf5441d41f3755326b9a3f62522eefb6 Jul 09 04:02:13.419713 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:13.419686 2571 generic.go:356] "Generic (PLEG): container finished" podID="85f28e4c-7cbb-4214-9095-e10b146b8c95" containerID="aa5a07e3eef4181cafbf6eb9ba9a86a77667fbb4e1fe0c6ec124d8a92c4e1499" exitCode=0 Jul 09 04:02:13.419825 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:13.419758 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" event={"ID":"85f28e4c-7cbb-4214-9095-e10b146b8c95","Type":"ContainerDied","Data":"aa5a07e3eef4181cafbf6eb9ba9a86a77667fbb4e1fe0c6ec124d8a92c4e1499"} Jul 09 04:02:13.419917 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:13.419837 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" event={"ID":"85f28e4c-7cbb-4214-9095-e10b146b8c95","Type":"ContainerStarted","Data":"30fe964659229a3a3d9d74614c2334bedf5441d41f3755326b9a3f62522eefb6"} Jul 09 04:02:13.421932 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:13.421907 2571 generic.go:356] "Generic (PLEG): container finished" podID="2924267d-1e47-42d4-8651-6fc55a0aaa22" containerID="a051af7e40b3dbc1facb21c9fb85f035785e3fdb579a577d99b39afe670f6c61" exitCode=0 Jul 09 04:02:13.422081 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:13.421981 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" event={"ID":"2924267d-1e47-42d4-8651-6fc55a0aaa22","Type":"ContainerDied","Data":"a051af7e40b3dbc1facb21c9fb85f035785e3fdb579a577d99b39afe670f6c61"} Jul 09 04:02:14.429784 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:14.429748 2571 generic.go:356] "Generic (PLEG): container finished" podID="85f28e4c-7cbb-4214-9095-e10b146b8c95" containerID="b4b3c9c7d8e89d90cee77dd735d6c7274b7ee2371bb9b3031b481eacf22493e9" exitCode=0 Jul 09 04:02:14.430259 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:14.429831 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" event={"ID":"85f28e4c-7cbb-4214-9095-e10b146b8c95","Type":"ContainerDied","Data":"b4b3c9c7d8e89d90cee77dd735d6c7274b7ee2371bb9b3031b481eacf22493e9"} Jul 09 04:02:14.432048 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:14.432030 2571 generic.go:356] "Generic (PLEG): container finished" podID="2924267d-1e47-42d4-8651-6fc55a0aaa22" containerID="f6463c5eca71c3983a52ef40f31d578dd4246a4e271d03e2bf34d180ecbb5f13" exitCode=0 Jul 09 04:02:14.432139 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:14.432115 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" event={"ID":"2924267d-1e47-42d4-8651-6fc55a0aaa22","Type":"ContainerDied","Data":"f6463c5eca71c3983a52ef40f31d578dd4246a4e271d03e2bf34d180ecbb5f13"} Jul 09 04:02:14.433742 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:14.433721 2571 generic.go:356] "Generic (PLEG): container finished" podID="77ed061d-f09c-4e24-8b76-ed05fef5630f" containerID="2e159db3f29eabfe365aa2eb78b1d4483b5cb46ff3595eff258f945ee6605f23" exitCode=0 Jul 09 04:02:14.433818 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:14.433773 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" event={"ID":"77ed061d-f09c-4e24-8b76-ed05fef5630f","Type":"ContainerDied","Data":"2e159db3f29eabfe365aa2eb78b1d4483b5cb46ff3595eff258f945ee6605f23"} Jul 09 04:02:14.438911 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:14.438888 2571 generic.go:356] "Generic (PLEG): container finished" podID="82053c64-4844-42ae-83e3-fab214a4f537" containerID="57bb2201d956f71ead714a114851ab706b281fc4424cbf983045dc7eefce69d8" exitCode=0 Jul 09 04:02:14.439004 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:14.438929 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" event={"ID":"82053c64-4844-42ae-83e3-fab214a4f537","Type":"ContainerDied","Data":"57bb2201d956f71ead714a114851ab706b281fc4424cbf983045dc7eefce69d8"} Jul 09 04:02:15.445905 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:15.445871 2571 generic.go:356] "Generic (PLEG): container finished" podID="77ed061d-f09c-4e24-8b76-ed05fef5630f" containerID="ba489f097f402033fdf878ceb35dbd8057304c5c2ef95209c39d20dce32d5416" exitCode=0 Jul 09 04:02:15.446357 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:15.445952 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" event={"ID":"77ed061d-f09c-4e24-8b76-ed05fef5630f","Type":"ContainerDied","Data":"ba489f097f402033fdf878ceb35dbd8057304c5c2ef95209c39d20dce32d5416"} Jul 09 04:02:15.447902 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:15.447882 2571 generic.go:356] "Generic (PLEG): container finished" podID="82053c64-4844-42ae-83e3-fab214a4f537" containerID="a1fcaa0d4094d53490329455d4ac442e1121946a07e03fa6040f9aaf149110b1" exitCode=0 Jul 09 04:02:15.448022 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:15.447953 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" event={"ID":"82053c64-4844-42ae-83e3-fab214a4f537","Type":"ContainerDied","Data":"a1fcaa0d4094d53490329455d4ac442e1121946a07e03fa6040f9aaf149110b1"} Jul 09 04:02:15.449895 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:15.449871 2571 generic.go:356] "Generic (PLEG): container finished" podID="85f28e4c-7cbb-4214-9095-e10b146b8c95" containerID="4b688e708006f1c6cd9d4780d6797441db6ab10aca9a363f284ae110e4569cd3" exitCode=0 Jul 09 04:02:15.449991 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:15.449940 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" event={"ID":"85f28e4c-7cbb-4214-9095-e10b146b8c95","Type":"ContainerDied","Data":"4b688e708006f1c6cd9d4780d6797441db6ab10aca9a363f284ae110e4569cd3"} Jul 09 04:02:15.582877 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:15.582832 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" Jul 09 04:02:15.670276 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:15.670246 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2924267d-1e47-42d4-8651-6fc55a0aaa22-util\") pod \"2924267d-1e47-42d4-8651-6fc55a0aaa22\" (UID: \"2924267d-1e47-42d4-8651-6fc55a0aaa22\") " Jul 09 04:02:15.670422 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:15.670369 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2924267d-1e47-42d4-8651-6fc55a0aaa22-bundle\") pod \"2924267d-1e47-42d4-8651-6fc55a0aaa22\" (UID: \"2924267d-1e47-42d4-8651-6fc55a0aaa22\") " Jul 09 04:02:15.670422 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:15.670387 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84lzt\" (UniqueName: \"kubernetes.io/projected/2924267d-1e47-42d4-8651-6fc55a0aaa22-kube-api-access-84lzt\") pod \"2924267d-1e47-42d4-8651-6fc55a0aaa22\" (UID: \"2924267d-1e47-42d4-8651-6fc55a0aaa22\") " Jul 09 04:02:15.670980 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:15.670949 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2924267d-1e47-42d4-8651-6fc55a0aaa22-bundle" (OuterVolumeSpecName: "bundle") pod "2924267d-1e47-42d4-8651-6fc55a0aaa22" (UID: "2924267d-1e47-42d4-8651-6fc55a0aaa22"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:02:15.672601 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:15.672576 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2924267d-1e47-42d4-8651-6fc55a0aaa22-kube-api-access-84lzt" (OuterVolumeSpecName: "kube-api-access-84lzt") pod "2924267d-1e47-42d4-8651-6fc55a0aaa22" (UID: "2924267d-1e47-42d4-8651-6fc55a0aaa22"). InnerVolumeSpecName "kube-api-access-84lzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:02:15.676337 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:15.676296 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2924267d-1e47-42d4-8651-6fc55a0aaa22-util" (OuterVolumeSpecName: "util") pod "2924267d-1e47-42d4-8651-6fc55a0aaa22" (UID: "2924267d-1e47-42d4-8651-6fc55a0aaa22"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:02:15.771324 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:15.771240 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2924267d-1e47-42d4-8651-6fc55a0aaa22-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:02:15.771324 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:15.771268 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84lzt\" (UniqueName: \"kubernetes.io/projected/2924267d-1e47-42d4-8651-6fc55a0aaa22-kube-api-access-84lzt\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:02:15.771324 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:15.771277 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2924267d-1e47-42d4-8651-6fc55a0aaa22-util\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:02:16.457257 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.457230 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" Jul 09 04:02:16.457691 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.457272 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx" event={"ID":"2924267d-1e47-42d4-8651-6fc55a0aaa22","Type":"ContainerDied","Data":"dcfcb369aa2e84c066ec35bcb08195d8f8d7a11ff0f98b7f9c9b8b406e9ed6f2"} Jul 09 04:02:16.457691 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.457339 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcfcb369aa2e84c066ec35bcb08195d8f8d7a11ff0f98b7f9c9b8b406e9ed6f2" Jul 09 04:02:16.614344 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.614323 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" Jul 09 04:02:16.616402 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.616385 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" Jul 09 04:02:16.657568 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.657539 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" Jul 09 04:02:16.677413 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.677375 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rkp6\" (UniqueName: \"kubernetes.io/projected/82053c64-4844-42ae-83e3-fab214a4f537-kube-api-access-6rkp6\") pod \"82053c64-4844-42ae-83e3-fab214a4f537\" (UID: \"82053c64-4844-42ae-83e3-fab214a4f537\") " Jul 09 04:02:16.677566 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.677421 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82053c64-4844-42ae-83e3-fab214a4f537-util\") pod \"82053c64-4844-42ae-83e3-fab214a4f537\" (UID: \"82053c64-4844-42ae-83e3-fab214a4f537\") " Jul 09 04:02:16.677566 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.677454 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnd9j\" (UniqueName: \"kubernetes.io/projected/85f28e4c-7cbb-4214-9095-e10b146b8c95-kube-api-access-hnd9j\") pod \"85f28e4c-7cbb-4214-9095-e10b146b8c95\" (UID: \"85f28e4c-7cbb-4214-9095-e10b146b8c95\") " Jul 09 04:02:16.677566 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.677483 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82053c64-4844-42ae-83e3-fab214a4f537-bundle\") pod \"82053c64-4844-42ae-83e3-fab214a4f537\" (UID: \"82053c64-4844-42ae-83e3-fab214a4f537\") " Jul 09 04:02:16.677566 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.677509 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77ed061d-f09c-4e24-8b76-ed05fef5630f-bundle\") pod \"77ed061d-f09c-4e24-8b76-ed05fef5630f\" (UID: \"77ed061d-f09c-4e24-8b76-ed05fef5630f\") " Jul 09 04:02:16.677566 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.677554 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7crx\" (UniqueName: \"kubernetes.io/projected/77ed061d-f09c-4e24-8b76-ed05fef5630f-kube-api-access-m7crx\") pod \"77ed061d-f09c-4e24-8b76-ed05fef5630f\" (UID: \"77ed061d-f09c-4e24-8b76-ed05fef5630f\") " Jul 09 04:02:16.677830 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.677598 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85f28e4c-7cbb-4214-9095-e10b146b8c95-bundle\") pod \"85f28e4c-7cbb-4214-9095-e10b146b8c95\" (UID: \"85f28e4c-7cbb-4214-9095-e10b146b8c95\") " Jul 09 04:02:16.677830 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.677644 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85f28e4c-7cbb-4214-9095-e10b146b8c95-util\") pod \"85f28e4c-7cbb-4214-9095-e10b146b8c95\" (UID: \"85f28e4c-7cbb-4214-9095-e10b146b8c95\") " Jul 09 04:02:16.677830 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.677679 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77ed061d-f09c-4e24-8b76-ed05fef5630f-util\") pod \"77ed061d-f09c-4e24-8b76-ed05fef5630f\" (UID: \"77ed061d-f09c-4e24-8b76-ed05fef5630f\") " Jul 09 04:02:16.678390 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.678356 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82053c64-4844-42ae-83e3-fab214a4f537-bundle" (OuterVolumeSpecName: "bundle") pod "82053c64-4844-42ae-83e3-fab214a4f537" (UID: "82053c64-4844-42ae-83e3-fab214a4f537"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:02:16.678568 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.678531 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ed061d-f09c-4e24-8b76-ed05fef5630f-bundle" (OuterVolumeSpecName: "bundle") pod "77ed061d-f09c-4e24-8b76-ed05fef5630f" (UID: "77ed061d-f09c-4e24-8b76-ed05fef5630f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:02:16.680044 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.679174 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f28e4c-7cbb-4214-9095-e10b146b8c95-bundle" (OuterVolumeSpecName: "bundle") pod "85f28e4c-7cbb-4214-9095-e10b146b8c95" (UID: "85f28e4c-7cbb-4214-9095-e10b146b8c95"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:02:16.680446 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.680418 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f28e4c-7cbb-4214-9095-e10b146b8c95-kube-api-access-hnd9j" (OuterVolumeSpecName: "kube-api-access-hnd9j") pod "85f28e4c-7cbb-4214-9095-e10b146b8c95" (UID: "85f28e4c-7cbb-4214-9095-e10b146b8c95"). InnerVolumeSpecName "kube-api-access-hnd9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:02:16.680676 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.680642 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ed061d-f09c-4e24-8b76-ed05fef5630f-kube-api-access-m7crx" (OuterVolumeSpecName: "kube-api-access-m7crx") pod "77ed061d-f09c-4e24-8b76-ed05fef5630f" (UID: "77ed061d-f09c-4e24-8b76-ed05fef5630f"). InnerVolumeSpecName "kube-api-access-m7crx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:02:16.681228 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.681207 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82053c64-4844-42ae-83e3-fab214a4f537-kube-api-access-6rkp6" (OuterVolumeSpecName: "kube-api-access-6rkp6") pod "82053c64-4844-42ae-83e3-fab214a4f537" (UID: "82053c64-4844-42ae-83e3-fab214a4f537"). InnerVolumeSpecName "kube-api-access-6rkp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:02:16.684758 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.684734 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ed061d-f09c-4e24-8b76-ed05fef5630f-util" (OuterVolumeSpecName: "util") pod "77ed061d-f09c-4e24-8b76-ed05fef5630f" (UID: "77ed061d-f09c-4e24-8b76-ed05fef5630f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:02:16.685630 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.685606 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82053c64-4844-42ae-83e3-fab214a4f537-util" (OuterVolumeSpecName: "util") pod "82053c64-4844-42ae-83e3-fab214a4f537" (UID: "82053c64-4844-42ae-83e3-fab214a4f537"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:02:16.686290 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.686271 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f28e4c-7cbb-4214-9095-e10b146b8c95-util" (OuterVolumeSpecName: "util") pod "85f28e4c-7cbb-4214-9095-e10b146b8c95" (UID: "85f28e4c-7cbb-4214-9095-e10b146b8c95"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:02:16.778654 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.778571 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82053c64-4844-42ae-83e3-fab214a4f537-util\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:02:16.778654 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.778600 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnd9j\" (UniqueName: \"kubernetes.io/projected/85f28e4c-7cbb-4214-9095-e10b146b8c95-kube-api-access-hnd9j\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:02:16.778654 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.778611 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82053c64-4844-42ae-83e3-fab214a4f537-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:02:16.778654 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.778619 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77ed061d-f09c-4e24-8b76-ed05fef5630f-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:02:16.778654 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.778627 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7crx\" (UniqueName: \"kubernetes.io/projected/77ed061d-f09c-4e24-8b76-ed05fef5630f-kube-api-access-m7crx\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:02:16.778654 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.778635 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85f28e4c-7cbb-4214-9095-e10b146b8c95-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:02:16.778654 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.778642 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85f28e4c-7cbb-4214-9095-e10b146b8c95-util\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:02:16.778654 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.778650 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77ed061d-f09c-4e24-8b76-ed05fef5630f-util\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:02:16.778654 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:16.778657 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rkp6\" (UniqueName: \"kubernetes.io/projected/82053c64-4844-42ae-83e3-fab214a4f537-kube-api-access-6rkp6\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:02:17.464661 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:17.464633 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" Jul 09 04:02:17.465161 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:17.464624 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn" event={"ID":"77ed061d-f09c-4e24-8b76-ed05fef5630f","Type":"ContainerDied","Data":"b51e4e1886d70834e3637e7e384b8284833b1ef83bb39cfb933baf11be5b9964"} Jul 09 04:02:17.465161 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:17.464753 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b51e4e1886d70834e3637e7e384b8284833b1ef83bb39cfb933baf11be5b9964" Jul 09 04:02:17.466409 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:17.466387 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" Jul 09 04:02:17.466409 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:17.466394 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4" event={"ID":"82053c64-4844-42ae-83e3-fab214a4f537","Type":"ContainerDied","Data":"ce8c9a2a4cd56da00206531a36cbe0ca4adbbd2b5abd6ebca30841ac04b428e9"} Jul 09 04:02:17.466579 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:17.466429 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce8c9a2a4cd56da00206531a36cbe0ca4adbbd2b5abd6ebca30841ac04b428e9" Jul 09 04:02:17.468251 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:17.468226 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" Jul 09 04:02:17.468337 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:17.468256 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2" event={"ID":"85f28e4c-7cbb-4214-9095-e10b146b8c95","Type":"ContainerDied","Data":"30fe964659229a3a3d9d74614c2334bedf5441d41f3755326b9a3f62522eefb6"} Jul 09 04:02:17.468337 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:02:17.468284 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30fe964659229a3a3d9d74614c2334bedf5441d41f3755326b9a3f62522eefb6" Jul 09 04:03:20.187121 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.187086 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d8fb6d944-82rp6"] Jul 09 04:03:20.188112 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188076 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2924267d-1e47-42d4-8651-6fc55a0aaa22" containerName="util" Jul 09 04:03:20.188112 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188105 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2924267d-1e47-42d4-8651-6fc55a0aaa22" containerName="util" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188125 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2924267d-1e47-42d4-8651-6fc55a0aaa22" containerName="extract" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188134 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2924267d-1e47-42d4-8651-6fc55a0aaa22" containerName="extract" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188145 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f28e4c-7cbb-4214-9095-e10b146b8c95" containerName="pull" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188153 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f28e4c-7cbb-4214-9095-e10b146b8c95" containerName="pull" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188162 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f28e4c-7cbb-4214-9095-e10b146b8c95" containerName="extract" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188170 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f28e4c-7cbb-4214-9095-e10b146b8c95" containerName="extract" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188181 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77ed061d-f09c-4e24-8b76-ed05fef5630f" containerName="extract" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188190 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="77ed061d-f09c-4e24-8b76-ed05fef5630f" containerName="extract" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188200 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2924267d-1e47-42d4-8651-6fc55a0aaa22" containerName="pull" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188207 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2924267d-1e47-42d4-8651-6fc55a0aaa22" containerName="pull" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188217 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82053c64-4844-42ae-83e3-fab214a4f537" containerName="pull" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188225 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="82053c64-4844-42ae-83e3-fab214a4f537" containerName="pull" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188232 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82053c64-4844-42ae-83e3-fab214a4f537" containerName="extract" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188239 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="82053c64-4844-42ae-83e3-fab214a4f537" containerName="extract" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188253 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f28e4c-7cbb-4214-9095-e10b146b8c95" containerName="util" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188261 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f28e4c-7cbb-4214-9095-e10b146b8c95" containerName="util" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188272 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82053c64-4844-42ae-83e3-fab214a4f537" containerName="util" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188279 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="82053c64-4844-42ae-83e3-fab214a4f537" containerName="util" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188290 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77ed061d-f09c-4e24-8b76-ed05fef5630f" containerName="pull" Jul 09 04:03:20.188288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188298 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="77ed061d-f09c-4e24-8b76-ed05fef5630f" containerName="pull" Jul 09 04:03:20.189087 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188313 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77ed061d-f09c-4e24-8b76-ed05fef5630f" containerName="util" Jul 09 04:03:20.189087 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188321 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="77ed061d-f09c-4e24-8b76-ed05fef5630f" containerName="util" Jul 09 04:03:20.189087 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188495 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2924267d-1e47-42d4-8651-6fc55a0aaa22" containerName="extract" Jul 09 04:03:20.189087 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188510 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="82053c64-4844-42ae-83e3-fab214a4f537" containerName="extract" Jul 09 04:03:20.189087 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188521 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="77ed061d-f09c-4e24-8b76-ed05fef5630f" containerName="extract" Jul 09 04:03:20.189087 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.188531 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f28e4c-7cbb-4214-9095-e10b146b8c95" containerName="extract" Jul 09 04:03:20.191720 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.191702 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.206165 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.206138 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d8fb6d944-82rp6"] Jul 09 04:03:20.221765 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.221742 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9725f88a-19c4-4203-aa62-2b33ea5feefd-trusted-ca-bundle\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.221896 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.221774 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9725f88a-19c4-4203-aa62-2b33ea5feefd-oauth-serving-cert\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.221896 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.221793 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvws\" (UniqueName: \"kubernetes.io/projected/9725f88a-19c4-4203-aa62-2b33ea5feefd-kube-api-access-zfvws\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.221896 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.221822 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9725f88a-19c4-4203-aa62-2b33ea5feefd-console-serving-cert\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.221896 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.221881 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9725f88a-19c4-4203-aa62-2b33ea5feefd-service-ca\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.222033 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.221913 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9725f88a-19c4-4203-aa62-2b33ea5feefd-console-config\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.222033 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.221945 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9725f88a-19c4-4203-aa62-2b33ea5feefd-console-oauth-config\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.323227 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.323184 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9725f88a-19c4-4203-aa62-2b33ea5feefd-trusted-ca-bundle\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.323227 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.323233 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9725f88a-19c4-4203-aa62-2b33ea5feefd-oauth-serving-cert\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.323442 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.323251 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvws\" (UniqueName: \"kubernetes.io/projected/9725f88a-19c4-4203-aa62-2b33ea5feefd-kube-api-access-zfvws\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.323442 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.323273 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9725f88a-19c4-4203-aa62-2b33ea5feefd-console-serving-cert\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.323442 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.323291 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9725f88a-19c4-4203-aa62-2b33ea5feefd-service-ca\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.323442 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.323325 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9725f88a-19c4-4203-aa62-2b33ea5feefd-console-config\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.323442 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.323357 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9725f88a-19c4-4203-aa62-2b33ea5feefd-console-oauth-config\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.324155 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.324123 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9725f88a-19c4-4203-aa62-2b33ea5feefd-oauth-serving-cert\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.324279 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.324175 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9725f88a-19c4-4203-aa62-2b33ea5feefd-trusted-ca-bundle\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.324279 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.324192 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9725f88a-19c4-4203-aa62-2b33ea5feefd-service-ca\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.324279 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.324192 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9725f88a-19c4-4203-aa62-2b33ea5feefd-console-config\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.325722 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.325702 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9725f88a-19c4-4203-aa62-2b33ea5feefd-console-serving-cert\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.325813 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.325733 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9725f88a-19c4-4203-aa62-2b33ea5feefd-console-oauth-config\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.330655 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.330631 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvws\" (UniqueName: \"kubernetes.io/projected/9725f88a-19c4-4203-aa62-2b33ea5feefd-kube-api-access-zfvws\") pod \"console-6d8fb6d944-82rp6\" (UID: \"9725f88a-19c4-4203-aa62-2b33ea5feefd\") " pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.502527 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.502456 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:20.632834 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.632813 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d8fb6d944-82rp6"] Jul 09 04:03:20.635119 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:03:20.635085 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9725f88a_19c4_4203_aa62_2b33ea5feefd.slice/crio-3e22b3b0c675ab01ccbdaba05c4efa8e5ec8d274ab6959cbc51ab83c2c04d6fa WatchSource:0}: Error finding container 3e22b3b0c675ab01ccbdaba05c4efa8e5ec8d274ab6959cbc51ab83c2c04d6fa: Status 404 returned error can't find the container with id 3e22b3b0c675ab01ccbdaba05c4efa8e5ec8d274ab6959cbc51ab83c2c04d6fa Jul 09 04:03:20.783660 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.783575 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d8fb6d944-82rp6" event={"ID":"9725f88a-19c4-4203-aa62-2b33ea5feefd","Type":"ContainerStarted","Data":"ee0f6ac869e29fa9b5a798fd0e5c3fef9ff5fb7bb68da03559769d2ff7a1dac6"} Jul 09 04:03:20.783660 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.783630 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d8fb6d944-82rp6" event={"ID":"9725f88a-19c4-4203-aa62-2b33ea5feefd","Type":"ContainerStarted","Data":"3e22b3b0c675ab01ccbdaba05c4efa8e5ec8d274ab6959cbc51ab83c2c04d6fa"} Jul 09 04:03:20.801635 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:20.801589 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d8fb6d944-82rp6" podStartSLOduration=0.801574073 podStartE2EDuration="801.574073ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:03:20.800486098 +0000 UTC m=+435.659357197" watchObservedRunningTime="2026-07-09 04:03:20.801574073 +0000 UTC m=+435.660445171" Jul 09 04:03:30.502672 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:30.502593 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:30.502672 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:30.502664 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:30.507389 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:30.507369 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:30.842362 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:30.842335 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d8fb6d944-82rp6" Jul 09 04:03:30.890257 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:30.890228 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c964f4cf9-lqrpk"] Jul 09 04:03:55.920011 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:55.919970 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-c964f4cf9-lqrpk" podUID="55a4799d-da99-4ece-a833-393d63d0d898" containerName="console" containerID="cri-o://cb3670f8bc86b7cad8bdd5c43fa3e4a643577a0ffd324d082e4a2c7bc75b1005" gracePeriod=15 Jul 09 04:03:56.172198 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.172143 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 04:03:56.260228 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.260196 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:03:56.260529 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.260518 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55a4799d-da99-4ece-a833-393d63d0d898" containerName="console" Jul 09 04:03:56.260586 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.260532 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="55a4799d-da99-4ece-a833-393d63d0d898" containerName="console" Jul 09 04:03:56.260621 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.260603 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="55a4799d-da99-4ece-a833-393d63d0d898" containerName="console" Jul 09 04:03:56.263415 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.263401 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-m6wqq" Jul 09 04:03:56.270574 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.270130 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 04:03:56.270574 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.270274 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 09 04:03:56.270574 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.270381 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 04:03:56.270574 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.270491 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-d4mf4\"" Jul 09 04:03:56.271921 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.271897 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:03:56.284351 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.284329 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:03:56.335325 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.335301 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55a4799d-da99-4ece-a833-393d63d0d898-console-serving-cert\") pod \"55a4799d-da99-4ece-a833-393d63d0d898\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " Jul 09 04:03:56.335429 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.335328 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55a4799d-da99-4ece-a833-393d63d0d898-console-oauth-config\") pod \"55a4799d-da99-4ece-a833-393d63d0d898\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " Jul 09 04:03:56.335429 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.335367 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fmnx\" (UniqueName: \"kubernetes.io/projected/55a4799d-da99-4ece-a833-393d63d0d898-kube-api-access-5fmnx\") pod \"55a4799d-da99-4ece-a833-393d63d0d898\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " Jul 09 04:03:56.335429 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.335389 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-trusted-ca-bundle\") pod \"55a4799d-da99-4ece-a833-393d63d0d898\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " Jul 09 04:03:56.335580 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.335566 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-service-ca\") pod \"55a4799d-da99-4ece-a833-393d63d0d898\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " Jul 09 04:03:56.335657 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.335633 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-oauth-serving-cert\") pod \"55a4799d-da99-4ece-a833-393d63d0d898\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " Jul 09 04:03:56.335795 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.335705 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-console-config\") pod \"55a4799d-da99-4ece-a833-393d63d0d898\" (UID: \"55a4799d-da99-4ece-a833-393d63d0d898\") " Jul 09 04:03:56.335895 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.335832 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "55a4799d-da99-4ece-a833-393d63d0d898" (UID: "55a4799d-da99-4ece-a833-393d63d0d898"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:03:56.335995 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.335959 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-service-ca" (OuterVolumeSpecName: "service-ca") pod "55a4799d-da99-4ece-a833-393d63d0d898" (UID: "55a4799d-da99-4ece-a833-393d63d0d898"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:03:56.335995 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.335966 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "55a4799d-da99-4ece-a833-393d63d0d898" (UID: "55a4799d-da99-4ece-a833-393d63d0d898"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:03:56.336229 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.336169 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-trusted-ca-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:03:56.336229 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.336190 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:03:56.336229 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.336207 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-oauth-serving-cert\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:03:56.336329 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.336226 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-console-config" (OuterVolumeSpecName: "console-config") pod "55a4799d-da99-4ece-a833-393d63d0d898" (UID: "55a4799d-da99-4ece-a833-393d63d0d898"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:03:56.337523 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.337497 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a4799d-da99-4ece-a833-393d63d0d898-kube-api-access-5fmnx" (OuterVolumeSpecName: "kube-api-access-5fmnx") pod "55a4799d-da99-4ece-a833-393d63d0d898" (UID: "55a4799d-da99-4ece-a833-393d63d0d898"). InnerVolumeSpecName "kube-api-access-5fmnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:03:56.337592 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.337523 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55a4799d-da99-4ece-a833-393d63d0d898-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "55a4799d-da99-4ece-a833-393d63d0d898" (UID: "55a4799d-da99-4ece-a833-393d63d0d898"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:03:56.337592 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.337544 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55a4799d-da99-4ece-a833-393d63d0d898-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "55a4799d-da99-4ece-a833-393d63d0d898" (UID: "55a4799d-da99-4ece-a833-393d63d0d898"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:03:56.437268 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.437204 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ac8490f-ff73-4272-8f68-782cfac6496f-config-file\") pod \"limitador-limitador-54d6d988d5-m6wqq\" (UID: \"3ac8490f-ff73-4272-8f68-782cfac6496f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m6wqq" Jul 09 04:03:56.437368 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.437274 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwff8\" (UniqueName: \"kubernetes.io/projected/3ac8490f-ff73-4272-8f68-782cfac6496f-kube-api-access-cwff8\") pod \"limitador-limitador-54d6d988d5-m6wqq\" (UID: \"3ac8490f-ff73-4272-8f68-782cfac6496f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m6wqq" Jul 09 04:03:56.437368 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.437303 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fmnx\" (UniqueName: \"kubernetes.io/projected/55a4799d-da99-4ece-a833-393d63d0d898-kube-api-access-5fmnx\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:03:56.437368 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.437313 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55a4799d-da99-4ece-a833-393d63d0d898-console-config\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:03:56.437368 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.437322 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55a4799d-da99-4ece-a833-393d63d0d898-console-serving-cert\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:03:56.437368 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.437330 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55a4799d-da99-4ece-a833-393d63d0d898-console-oauth-config\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:03:56.537774 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.537745 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwff8\" (UniqueName: \"kubernetes.io/projected/3ac8490f-ff73-4272-8f68-782cfac6496f-kube-api-access-cwff8\") pod \"limitador-limitador-54d6d988d5-m6wqq\" (UID: \"3ac8490f-ff73-4272-8f68-782cfac6496f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m6wqq" Jul 09 04:03:56.537919 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.537790 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ac8490f-ff73-4272-8f68-782cfac6496f-config-file\") pod \"limitador-limitador-54d6d988d5-m6wqq\" (UID: \"3ac8490f-ff73-4272-8f68-782cfac6496f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m6wqq" Jul 09 04:03:56.538359 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.538341 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ac8490f-ff73-4272-8f68-782cfac6496f-config-file\") pod \"limitador-limitador-54d6d988d5-m6wqq\" (UID: \"3ac8490f-ff73-4272-8f68-782cfac6496f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m6wqq" Jul 09 04:03:56.549387 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.549363 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwff8\" (UniqueName: \"kubernetes.io/projected/3ac8490f-ff73-4272-8f68-782cfac6496f-kube-api-access-cwff8\") pod \"limitador-limitador-54d6d988d5-m6wqq\" (UID: \"3ac8490f-ff73-4272-8f68-782cfac6496f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m6wqq" Jul 09 04:03:56.574222 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.574200 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-m6wqq" Jul 09 04:03:56.704475 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:03:56.704439 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac8490f_ff73_4272_8f68_782cfac6496f.slice/crio-99f28e2fc3b80efdf1435c287ec39b47c6005ae4ffb6c3bb3257e4f7053df659 WatchSource:0}: Error finding container 99f28e2fc3b80efdf1435c287ec39b47c6005ae4ffb6c3bb3257e4f7053df659: Status 404 returned error can't find the container with id 99f28e2fc3b80efdf1435c287ec39b47c6005ae4ffb6c3bb3257e4f7053df659 Jul 09 04:03:56.709537 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.709092 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:03:56.830879 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.830837 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-2mrfd"] Jul 09 04:03:56.835513 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.835493 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-2mrfd" Jul 09 04:03:56.841248 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.841226 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-2mrfd"] Jul 09 04:03:56.841755 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.841732 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kgtrg\"" Jul 09 04:03:56.941068 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.941041 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9kz6\" (UniqueName: \"kubernetes.io/projected/9bed3206-96da-4d40-8b90-f2e3f289b6da-kube-api-access-x9kz6\") pod \"authorino-58cd955675-2mrfd\" (UID: \"9bed3206-96da-4d40-8b90-f2e3f289b6da\") " pod="kuadrant-system/authorino-58cd955675-2mrfd" Jul 09 04:03:56.974293 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.974221 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-m6wqq" event={"ID":"3ac8490f-ff73-4272-8f68-782cfac6496f","Type":"ContainerStarted","Data":"99f28e2fc3b80efdf1435c287ec39b47c6005ae4ffb6c3bb3257e4f7053df659"} Jul 09 04:03:56.975472 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.975450 2571 generic.go:356] "Generic (PLEG): container finished" podID="55a4799d-da99-4ece-a833-393d63d0d898" containerID="cb3670f8bc86b7cad8bdd5c43fa3e4a643577a0ffd324d082e4a2c7bc75b1005" exitCode=0 Jul 09 04:03:56.975546 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.975487 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c964f4cf9-lqrpk" event={"ID":"55a4799d-da99-4ece-a833-393d63d0d898","Type":"ContainerDied","Data":"cb3670f8bc86b7cad8bdd5c43fa3e4a643577a0ffd324d082e4a2c7bc75b1005"} Jul 09 04:03:56.975546 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.975519 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c964f4cf9-lqrpk" event={"ID":"55a4799d-da99-4ece-a833-393d63d0d898","Type":"ContainerDied","Data":"bed4e1681326b0e8c220aa7135a87774e532d4da0f66192e106a4f6bc4551978"} Jul 09 04:03:56.975546 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.975517 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c964f4cf9-lqrpk" Jul 09 04:03:56.975658 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.975532 2571 scope.go:117] "RemoveContainer" containerID="cb3670f8bc86b7cad8bdd5c43fa3e4a643577a0ffd324d082e4a2c7bc75b1005" Jul 09 04:03:56.985167 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.985151 2571 scope.go:117] "RemoveContainer" containerID="cb3670f8bc86b7cad8bdd5c43fa3e4a643577a0ffd324d082e4a2c7bc75b1005" Jul 09 04:03:56.985419 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:03:56.985403 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb3670f8bc86b7cad8bdd5c43fa3e4a643577a0ffd324d082e4a2c7bc75b1005\": container with ID starting with cb3670f8bc86b7cad8bdd5c43fa3e4a643577a0ffd324d082e4a2c7bc75b1005 not found: ID does not exist" containerID="cb3670f8bc86b7cad8bdd5c43fa3e4a643577a0ffd324d082e4a2c7bc75b1005" Jul 09 04:03:56.985464 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:56.985427 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb3670f8bc86b7cad8bdd5c43fa3e4a643577a0ffd324d082e4a2c7bc75b1005"} err="failed to get container status \"cb3670f8bc86b7cad8bdd5c43fa3e4a643577a0ffd324d082e4a2c7bc75b1005\": rpc error: code = NotFound desc = could not find container \"cb3670f8bc86b7cad8bdd5c43fa3e4a643577a0ffd324d082e4a2c7bc75b1005\": container with ID starting with cb3670f8bc86b7cad8bdd5c43fa3e4a643577a0ffd324d082e4a2c7bc75b1005 not found: ID does not exist" Jul 09 04:03:57.005210 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:57.005187 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c964f4cf9-lqrpk"] Jul 09 04:03:57.010050 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:57.010034 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c964f4cf9-lqrpk"] Jul 09 04:03:57.042363 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:57.042336 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9kz6\" (UniqueName: \"kubernetes.io/projected/9bed3206-96da-4d40-8b90-f2e3f289b6da-kube-api-access-x9kz6\") pod \"authorino-58cd955675-2mrfd\" (UID: \"9bed3206-96da-4d40-8b90-f2e3f289b6da\") " pod="kuadrant-system/authorino-58cd955675-2mrfd" Jul 09 04:03:57.053517 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:57.053500 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9kz6\" (UniqueName: \"kubernetes.io/projected/9bed3206-96da-4d40-8b90-f2e3f289b6da-kube-api-access-x9kz6\") pod \"authorino-58cd955675-2mrfd\" (UID: \"9bed3206-96da-4d40-8b90-f2e3f289b6da\") " pod="kuadrant-system/authorino-58cd955675-2mrfd" Jul 09 04:03:57.146514 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:57.146488 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-2mrfd" Jul 09 04:03:57.314728 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:57.314671 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-2mrfd"] Jul 09 04:03:57.320355 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:03:57.320322 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bed3206_96da_4d40_8b90_f2e3f289b6da.slice/crio-711c42960986bc7dcd8e6591a5d9f6910db46560049f539ba3f7c86af07b59bf WatchSource:0}: Error finding container 711c42960986bc7dcd8e6591a5d9f6910db46560049f539ba3f7c86af07b59bf: Status 404 returned error can't find the container with id 711c42960986bc7dcd8e6591a5d9f6910db46560049f539ba3f7c86af07b59bf Jul 09 04:03:57.630631 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:57.630597 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55a4799d-da99-4ece-a833-393d63d0d898" path="/var/lib/kubelet/pods/55a4799d-da99-4ece-a833-393d63d0d898/volumes" Jul 09 04:03:57.986520 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:03:57.986435 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-2mrfd" event={"ID":"9bed3206-96da-4d40-8b90-f2e3f289b6da","Type":"ContainerStarted","Data":"711c42960986bc7dcd8e6591a5d9f6910db46560049f539ba3f7c86af07b59bf"} Jul 09 04:04:01.421634 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:01.421601 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-2mrfd"] Jul 09 04:04:02.010687 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:02.010654 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-m6wqq" event={"ID":"3ac8490f-ff73-4272-8f68-782cfac6496f","Type":"ContainerStarted","Data":"c0f85368766200d3b909c2c232457e9fac93f69c67c3df1a32d0051393eb0867"} Jul 09 04:04:02.010900 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:02.010870 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-m6wqq" Jul 09 04:04:02.012088 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:02.012065 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-2mrfd" event={"ID":"9bed3206-96da-4d40-8b90-f2e3f289b6da","Type":"ContainerStarted","Data":"3a717c5de61919a474d822dfcd535f11db968fca6e42e8206aa898636fe2a13a"} Jul 09 04:04:02.033421 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:02.033381 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-m6wqq" podStartSLOduration=1.667458564 podStartE2EDuration="6.033367714s" totalImagesPullingTime="4.36590915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:04:02.031381456 +0000 UTC m=+476.890252556" watchObservedRunningTime="2026-07-09 04:04:02.033367714 +0000 UTC m=+476.892238827" Jul 09 04:04:02.053118 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:02.053076 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-2mrfd" podStartSLOduration=2.342169129 podStartE2EDuration="6.053061946s" totalImagesPullingTime="3.710892817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:04:02.05081882 +0000 UTC m=+476.909689920" watchObservedRunningTime="2026-07-09 04:04:02.053061946 +0000 UTC m=+476.911933046" Jul 09 04:04:03.017387 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:03.017346 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-2mrfd" podUID="9bed3206-96da-4d40-8b90-f2e3f289b6da" containerName="authorino" containerID="cri-o://3a717c5de61919a474d822dfcd535f11db968fca6e42e8206aa898636fe2a13a" gracePeriod=30 Jul 09 04:04:03.264206 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:03.264184 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-2mrfd" Jul 09 04:04:03.392302 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:03.392273 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9kz6\" (UniqueName: \"kubernetes.io/projected/9bed3206-96da-4d40-8b90-f2e3f289b6da-kube-api-access-x9kz6\") pod \"9bed3206-96da-4d40-8b90-f2e3f289b6da\" (UID: \"9bed3206-96da-4d40-8b90-f2e3f289b6da\") " Jul 09 04:04:03.394225 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:03.394191 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bed3206-96da-4d40-8b90-f2e3f289b6da-kube-api-access-x9kz6" (OuterVolumeSpecName: "kube-api-access-x9kz6") pod "9bed3206-96da-4d40-8b90-f2e3f289b6da" (UID: "9bed3206-96da-4d40-8b90-f2e3f289b6da"). InnerVolumeSpecName "kube-api-access-x9kz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:04:03.493196 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:03.493170 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9kz6\" (UniqueName: \"kubernetes.io/projected/9bed3206-96da-4d40-8b90-f2e3f289b6da-kube-api-access-x9kz6\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:04:04.023782 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:04.023752 2571 generic.go:356] "Generic (PLEG): container finished" podID="9bed3206-96da-4d40-8b90-f2e3f289b6da" containerID="3a717c5de61919a474d822dfcd535f11db968fca6e42e8206aa898636fe2a13a" exitCode=0 Jul 09 04:04:04.024322 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:04.023791 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-2mrfd" Jul 09 04:04:04.024322 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:04.023827 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-2mrfd" event={"ID":"9bed3206-96da-4d40-8b90-f2e3f289b6da","Type":"ContainerDied","Data":"3a717c5de61919a474d822dfcd535f11db968fca6e42e8206aa898636fe2a13a"} Jul 09 04:04:04.024322 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:04.023892 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-2mrfd" event={"ID":"9bed3206-96da-4d40-8b90-f2e3f289b6da","Type":"ContainerDied","Data":"711c42960986bc7dcd8e6591a5d9f6910db46560049f539ba3f7c86af07b59bf"} Jul 09 04:04:04.024322 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:04.023917 2571 scope.go:117] "RemoveContainer" containerID="3a717c5de61919a474d822dfcd535f11db968fca6e42e8206aa898636fe2a13a" Jul 09 04:04:04.032436 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:04.032401 2571 scope.go:117] "RemoveContainer" containerID="3a717c5de61919a474d822dfcd535f11db968fca6e42e8206aa898636fe2a13a" Jul 09 04:04:04.032664 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:04:04.032647 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a717c5de61919a474d822dfcd535f11db968fca6e42e8206aa898636fe2a13a\": container with ID starting with 3a717c5de61919a474d822dfcd535f11db968fca6e42e8206aa898636fe2a13a not found: ID does not exist" containerID="3a717c5de61919a474d822dfcd535f11db968fca6e42e8206aa898636fe2a13a" Jul 09 04:04:04.032715 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:04.032673 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a717c5de61919a474d822dfcd535f11db968fca6e42e8206aa898636fe2a13a"} err="failed to get container status \"3a717c5de61919a474d822dfcd535f11db968fca6e42e8206aa898636fe2a13a\": rpc error: code = NotFound desc = could not find container \"3a717c5de61919a474d822dfcd535f11db968fca6e42e8206aa898636fe2a13a\": container with ID starting with 3a717c5de61919a474d822dfcd535f11db968fca6e42e8206aa898636fe2a13a not found: ID does not exist" Jul 09 04:04:04.038430 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:04.038409 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-2mrfd"] Jul 09 04:04:04.042167 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:04.042145 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-2mrfd"] Jul 09 04:04:05.631810 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:05.631774 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bed3206-96da-4d40-8b90-f2e3f289b6da" path="/var/lib/kubelet/pods/9bed3206-96da-4d40-8b90-f2e3f289b6da/volumes" Jul 09 04:04:13.018020 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:13.017989 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-m6wqq" Jul 09 04:04:27.066435 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.066397 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tbhsp"] Jul 09 04:04:27.066929 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.066793 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bed3206-96da-4d40-8b90-f2e3f289b6da" containerName="authorino" Jul 09 04:04:27.066929 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.066805 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bed3206-96da-4d40-8b90-f2e3f289b6da" containerName="authorino" Jul 09 04:04:27.066929 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.066907 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bed3206-96da-4d40-8b90-f2e3f289b6da" containerName="authorino" Jul 09 04:04:27.074575 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.074542 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tbhsp" Jul 09 04:04:27.075028 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.075001 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tbhsp"] Jul 09 04:04:27.077183 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.077152 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kgtrg\"" Jul 09 04:04:27.185757 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.185727 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh2rj\" (UniqueName: \"kubernetes.io/projected/82ef31d7-dbba-46ee-ae4b-26ab0640f20e-kube-api-access-kh2rj\") pod \"authorino-5d4b67c446-tbhsp\" (UID: \"82ef31d7-dbba-46ee-ae4b-26ab0640f20e\") " pod="kuadrant-system/authorino-5d4b67c446-tbhsp" Jul 09 04:04:27.285879 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.285834 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tbhsp"] Jul 09 04:04:27.286111 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:04:27.286089 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-kh2rj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-tbhsp" podUID="82ef31d7-dbba-46ee-ae4b-26ab0640f20e" Jul 09 04:04:27.286202 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.286178 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh2rj\" (UniqueName: \"kubernetes.io/projected/82ef31d7-dbba-46ee-ae4b-26ab0640f20e-kube-api-access-kh2rj\") pod \"authorino-5d4b67c446-tbhsp\" (UID: \"82ef31d7-dbba-46ee-ae4b-26ab0640f20e\") " pod="kuadrant-system/authorino-5d4b67c446-tbhsp" Jul 09 04:04:27.299701 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.299668 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh2rj\" (UniqueName: \"kubernetes.io/projected/82ef31d7-dbba-46ee-ae4b-26ab0640f20e-kube-api-access-kh2rj\") pod \"authorino-5d4b67c446-tbhsp\" (UID: \"82ef31d7-dbba-46ee-ae4b-26ab0640f20e\") " pod="kuadrant-system/authorino-5d4b67c446-tbhsp" Jul 09 04:04:27.430431 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.430394 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dbd6b8ff4-52f2p"] Jul 09 04:04:27.432961 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.432941 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dbd6b8ff4-52f2p" Jul 09 04:04:27.435152 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.435130 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 09 04:04:27.439932 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.439906 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dbd6b8ff4-52f2p"] Jul 09 04:04:27.488180 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.488151 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a00e7bb-8155-4318-9959-2d488a6d1d79-tls-cert\") pod \"authorino-5dbd6b8ff4-52f2p\" (UID: \"0a00e7bb-8155-4318-9959-2d488a6d1d79\") " pod="kuadrant-system/authorino-5dbd6b8ff4-52f2p" Jul 09 04:04:27.488369 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.488193 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf5kl\" (UniqueName: \"kubernetes.io/projected/0a00e7bb-8155-4318-9959-2d488a6d1d79-kube-api-access-vf5kl\") pod \"authorino-5dbd6b8ff4-52f2p\" (UID: \"0a00e7bb-8155-4318-9959-2d488a6d1d79\") " pod="kuadrant-system/authorino-5dbd6b8ff4-52f2p" Jul 09 04:04:27.589510 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.589457 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a00e7bb-8155-4318-9959-2d488a6d1d79-tls-cert\") pod \"authorino-5dbd6b8ff4-52f2p\" (UID: \"0a00e7bb-8155-4318-9959-2d488a6d1d79\") " pod="kuadrant-system/authorino-5dbd6b8ff4-52f2p" Jul 09 04:04:27.589683 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.589523 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf5kl\" (UniqueName: \"kubernetes.io/projected/0a00e7bb-8155-4318-9959-2d488a6d1d79-kube-api-access-vf5kl\") pod \"authorino-5dbd6b8ff4-52f2p\" (UID: \"0a00e7bb-8155-4318-9959-2d488a6d1d79\") " pod="kuadrant-system/authorino-5dbd6b8ff4-52f2p" Jul 09 04:04:27.591988 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.591962 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a00e7bb-8155-4318-9959-2d488a6d1d79-tls-cert\") pod \"authorino-5dbd6b8ff4-52f2p\" (UID: \"0a00e7bb-8155-4318-9959-2d488a6d1d79\") " pod="kuadrant-system/authorino-5dbd6b8ff4-52f2p" Jul 09 04:04:27.596796 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.596772 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf5kl\" (UniqueName: \"kubernetes.io/projected/0a00e7bb-8155-4318-9959-2d488a6d1d79-kube-api-access-vf5kl\") pod \"authorino-5dbd6b8ff4-52f2p\" (UID: \"0a00e7bb-8155-4318-9959-2d488a6d1d79\") " pod="kuadrant-system/authorino-5dbd6b8ff4-52f2p" Jul 09 04:04:27.743426 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.743342 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dbd6b8ff4-52f2p" Jul 09 04:04:27.872693 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:27.872662 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dbd6b8ff4-52f2p"] Jul 09 04:04:27.875966 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:04:27.875933 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a00e7bb_8155_4318_9959_2d488a6d1d79.slice/crio-16b632a52f0a2509bd83ab4272a00d920dc4ce8358c2bf421e9ec1de6f75a051 WatchSource:0}: Error finding container 16b632a52f0a2509bd83ab4272a00d920dc4ce8358c2bf421e9ec1de6f75a051: Status 404 returned error can't find the container with id 16b632a52f0a2509bd83ab4272a00d920dc4ce8358c2bf421e9ec1de6f75a051 Jul 09 04:04:28.145798 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:28.145773 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tbhsp" Jul 09 04:04:28.145798 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:28.145773 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dbd6b8ff4-52f2p" event={"ID":"0a00e7bb-8155-4318-9959-2d488a6d1d79","Type":"ContainerStarted","Data":"16b632a52f0a2509bd83ab4272a00d920dc4ce8358c2bf421e9ec1de6f75a051"} Jul 09 04:04:28.151729 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:28.151708 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tbhsp" Jul 09 04:04:28.195051 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:28.195014 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh2rj\" (UniqueName: \"kubernetes.io/projected/82ef31d7-dbba-46ee-ae4b-26ab0640f20e-kube-api-access-kh2rj\") pod \"82ef31d7-dbba-46ee-ae4b-26ab0640f20e\" (UID: \"82ef31d7-dbba-46ee-ae4b-26ab0640f20e\") " Jul 09 04:04:28.197119 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:28.197094 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ef31d7-dbba-46ee-ae4b-26ab0640f20e-kube-api-access-kh2rj" (OuterVolumeSpecName: "kube-api-access-kh2rj") pod "82ef31d7-dbba-46ee-ae4b-26ab0640f20e" (UID: "82ef31d7-dbba-46ee-ae4b-26ab0640f20e"). InnerVolumeSpecName "kube-api-access-kh2rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:04:28.296265 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:28.296221 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh2rj\" (UniqueName: \"kubernetes.io/projected/82ef31d7-dbba-46ee-ae4b-26ab0640f20e-kube-api-access-kh2rj\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:04:29.151944 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:29.151900 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dbd6b8ff4-52f2p" event={"ID":"0a00e7bb-8155-4318-9959-2d488a6d1d79","Type":"ContainerStarted","Data":"cb70437fa4472d7989ed879179edc2019fd17cbb0118046152fc4fe16726f173"} Jul 09 04:04:29.152424 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:29.151953 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tbhsp" Jul 09 04:04:29.168545 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:29.168496 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dbd6b8ff4-52f2p" podStartSLOduration=1.679553168 podStartE2EDuration="2.168479045s" totalImagesPullingTime="488.925877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:04:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:04:29.165631478 +0000 UTC m=+504.024502592" watchObservedRunningTime="2026-07-09 04:04:29.168479045 +0000 UTC m=+504.027350154" Jul 09 04:04:29.187627 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:29.187597 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tbhsp"] Jul 09 04:04:29.197966 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:29.197931 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tbhsp"] Jul 09 04:04:29.626798 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:04:29.626761 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82ef31d7-dbba-46ee-ae4b-26ab0640f20e" path="/var/lib/kubelet/pods/82ef31d7-dbba-46ee-ae4b-26ab0640f20e/volumes" Jul 09 04:05:10.770315 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:05:10.770223 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:05:33.033766 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:05:33.033727 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:05:41.126111 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:05:41.126066 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:05:50.536041 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:05:50.536006 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:05:55.036741 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:05:55.036705 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:06:05.557143 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:05.557111 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:06:05.559405 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:05.559385 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:06:11.720935 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.720896 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28"] Jul 09 04:06:11.724421 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.724405 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.726671 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.726652 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 04:06:11.726791 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.726677 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lmz6c\"" Jul 09 04:06:11.727464 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.727449 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 04:06:11.727464 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.727458 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 09 04:06:11.734129 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.734109 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28"] Jul 09 04:06:11.774839 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.774806 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a80b4b0-6524-4def-9527-c7ece21d4497-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.774839 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.774843 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a80b4b0-6524-4def-9527-c7ece21d4497-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.775037 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.774953 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a80b4b0-6524-4def-9527-c7ece21d4497-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.775037 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.774979 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjcl\" (UniqueName: \"kubernetes.io/projected/8a80b4b0-6524-4def-9527-c7ece21d4497-kube-api-access-pqjcl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.775037 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.775013 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a80b4b0-6524-4def-9527-c7ece21d4497-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.775138 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.775081 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a80b4b0-6524-4def-9527-c7ece21d4497-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.875908 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.875876 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a80b4b0-6524-4def-9527-c7ece21d4497-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.876064 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.875925 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a80b4b0-6524-4def-9527-c7ece21d4497-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.876064 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.875990 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a80b4b0-6524-4def-9527-c7ece21d4497-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.876064 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.876015 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjcl\" (UniqueName: \"kubernetes.io/projected/8a80b4b0-6524-4def-9527-c7ece21d4497-kube-api-access-pqjcl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.876184 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.876062 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a80b4b0-6524-4def-9527-c7ece21d4497-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.876184 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.876112 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a80b4b0-6524-4def-9527-c7ece21d4497-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.876343 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.876321 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a80b4b0-6524-4def-9527-c7ece21d4497-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.876409 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.876344 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a80b4b0-6524-4def-9527-c7ece21d4497-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.876464 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.876412 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a80b4b0-6524-4def-9527-c7ece21d4497-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.878239 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.878220 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a80b4b0-6524-4def-9527-c7ece21d4497-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.878417 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.878398 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a80b4b0-6524-4def-9527-c7ece21d4497-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:11.885987 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:11.885961 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjcl\" (UniqueName: \"kubernetes.io/projected/8a80b4b0-6524-4def-9527-c7ece21d4497-kube-api-access-pqjcl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fbs28\" (UID: \"8a80b4b0-6524-4def-9527-c7ece21d4497\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:12.035034 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:12.034966 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:12.164377 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:12.164344 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28"] Jul 09 04:06:12.165946 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:06:12.165921 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a80b4b0_6524_4def_9527_c7ece21d4497.slice/crio-f7c321d76b797bbdc470b86385033d307350fd41fe744eaf5e3e4d1b39f55184 WatchSource:0}: Error finding container f7c321d76b797bbdc470b86385033d307350fd41fe744eaf5e3e4d1b39f55184: Status 404 returned error can't find the container with id f7c321d76b797bbdc470b86385033d307350fd41fe744eaf5e3e4d1b39f55184 Jul 09 04:06:12.692527 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:12.692495 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" event={"ID":"8a80b4b0-6524-4def-9527-c7ece21d4497","Type":"ContainerStarted","Data":"f7c321d76b797bbdc470b86385033d307350fd41fe744eaf5e3e4d1b39f55184"} Jul 09 04:06:13.637122 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:13.637093 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:06:13.700309 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:13.700275 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" event={"ID":"8a80b4b0-6524-4def-9527-c7ece21d4497","Type":"ContainerStarted","Data":"3ddcf8a9dafc2986df054d5ca1842ae64236b07a3ed68ba729814a7e1a796c73"} Jul 09 04:06:13.719003 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:13.718952 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" podStartSLOduration=1.694943111 podStartE2EDuration="2.718936572s" totalImagesPullingTime="1.023993461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:06:13.716832392 +0000 UTC m=+608.575703494" watchObservedRunningTime="2026-07-09 04:06:13.718936572 +0000 UTC m=+608.577807671" Jul 09 04:06:21.941734 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:21.941700 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc"] Jul 09 04:06:21.945101 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:21.945084 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:21.947205 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:21.947186 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 09 04:06:21.953403 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:21.953378 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc"] Jul 09 04:06:22.035665 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.035634 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:22.035813 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.035685 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:22.047700 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.047677 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:22.062993 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.062971 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpqjs\" (UniqueName: \"kubernetes.io/projected/dad24a98-995c-459b-b85d-7bf18ff5a3c8-kube-api-access-zpqjs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.063081 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.063008 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad24a98-995c-459b-b85d-7bf18ff5a3c8-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.063123 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.063088 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad24a98-995c-459b-b85d-7bf18ff5a3c8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.063159 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.063120 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad24a98-995c-459b-b85d-7bf18ff5a3c8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.063159 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.063145 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad24a98-995c-459b-b85d-7bf18ff5a3c8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.063223 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.063187 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad24a98-995c-459b-b85d-7bf18ff5a3c8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.163980 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.163952 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpqjs\" (UniqueName: \"kubernetes.io/projected/dad24a98-995c-459b-b85d-7bf18ff5a3c8-kube-api-access-zpqjs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.164114 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.164000 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad24a98-995c-459b-b85d-7bf18ff5a3c8-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.164114 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.164070 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad24a98-995c-459b-b85d-7bf18ff5a3c8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.164114 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.164096 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad24a98-995c-459b-b85d-7bf18ff5a3c8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.164273 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.164242 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad24a98-995c-459b-b85d-7bf18ff5a3c8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.164331 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.164317 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad24a98-995c-459b-b85d-7bf18ff5a3c8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.164498 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.164478 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad24a98-995c-459b-b85d-7bf18ff5a3c8-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.164558 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.164525 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad24a98-995c-459b-b85d-7bf18ff5a3c8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.164603 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.164581 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad24a98-995c-459b-b85d-7bf18ff5a3c8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.166376 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.166355 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad24a98-995c-459b-b85d-7bf18ff5a3c8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.166620 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.166604 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad24a98-995c-459b-b85d-7bf18ff5a3c8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.171337 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.171314 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpqjs\" (UniqueName: \"kubernetes.io/projected/dad24a98-995c-459b-b85d-7bf18ff5a3c8-kube-api-access-zpqjs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc\" (UID: \"dad24a98-995c-459b-b85d-7bf18ff5a3c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.255322 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.255257 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:22.386362 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.386334 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc"] Jul 09 04:06:22.387507 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:06:22.387486 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddad24a98_995c_459b_b85d_7bf18ff5a3c8.slice/crio-347067d71ebe596c061a190c0de93a21ac71a9d4ff703f2f5600cdbb3ca9413e WatchSource:0}: Error finding container 347067d71ebe596c061a190c0de93a21ac71a9d4ff703f2f5600cdbb3ca9413e: Status 404 returned error can't find the container with id 347067d71ebe596c061a190c0de93a21ac71a9d4ff703f2f5600cdbb3ca9413e Jul 09 04:06:22.748269 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.748230 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" event={"ID":"dad24a98-995c-459b-b85d-7bf18ff5a3c8","Type":"ContainerStarted","Data":"6d165b62e6c1fea6b3a617160c5939564fd977e6ba3a83b684066dc3930353cb"} Jul 09 04:06:22.748429 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.748282 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" event={"ID":"dad24a98-995c-459b-b85d-7bf18ff5a3c8","Type":"ContainerStarted","Data":"347067d71ebe596c061a190c0de93a21ac71a9d4ff703f2f5600cdbb3ca9413e"} Jul 09 04:06:22.759725 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.759700 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fbs28" Jul 09 04:06:22.765831 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:22.765791 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" podStartSLOduration=1.603058502 podStartE2EDuration="1.765778123s" totalImagesPullingTime="162.719621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:06:22.76369695 +0000 UTC m=+617.622568049" watchObservedRunningTime="2026-07-09 04:06:22.765778123 +0000 UTC m=+617.624649223" Jul 09 04:06:24.938871 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:24.938833 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:06:32.256105 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:32.256021 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:32.256105 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:32.256072 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:32.268399 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:32.268377 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:06:32.810572 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:06:32.810545 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc" Jul 09 04:07:31.546188 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:31.546153 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dbd6b8ff4-52f2p"] Jul 09 04:07:31.546712 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:31.546389 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5dbd6b8ff4-52f2p" podUID="0a00e7bb-8155-4318-9959-2d488a6d1d79" containerName="authorino" containerID="cri-o://cb70437fa4472d7989ed879179edc2019fd17cbb0118046152fc4fe16726f173" gracePeriod=30 Jul 09 04:07:31.781986 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:31.781964 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dbd6b8ff4-52f2p" Jul 09 04:07:31.849995 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:31.849963 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a00e7bb-8155-4318-9959-2d488a6d1d79-tls-cert\") pod \"0a00e7bb-8155-4318-9959-2d488a6d1d79\" (UID: \"0a00e7bb-8155-4318-9959-2d488a6d1d79\") " Jul 09 04:07:31.850148 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:31.850054 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf5kl\" (UniqueName: \"kubernetes.io/projected/0a00e7bb-8155-4318-9959-2d488a6d1d79-kube-api-access-vf5kl\") pod \"0a00e7bb-8155-4318-9959-2d488a6d1d79\" (UID: \"0a00e7bb-8155-4318-9959-2d488a6d1d79\") " Jul 09 04:07:31.852013 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:31.851984 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a00e7bb-8155-4318-9959-2d488a6d1d79-kube-api-access-vf5kl" (OuterVolumeSpecName: "kube-api-access-vf5kl") pod "0a00e7bb-8155-4318-9959-2d488a6d1d79" (UID: "0a00e7bb-8155-4318-9959-2d488a6d1d79"). InnerVolumeSpecName "kube-api-access-vf5kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:07:31.859874 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:31.859837 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a00e7bb-8155-4318-9959-2d488a6d1d79-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "0a00e7bb-8155-4318-9959-2d488a6d1d79" (UID: "0a00e7bb-8155-4318-9959-2d488a6d1d79"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:07:31.951087 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:31.951059 2571 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a00e7bb-8155-4318-9959-2d488a6d1d79-tls-cert\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:07:31.951087 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:31.951084 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf5kl\" (UniqueName: \"kubernetes.io/projected/0a00e7bb-8155-4318-9959-2d488a6d1d79-kube-api-access-vf5kl\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:07:32.109059 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:32.108941 2571 generic.go:356] "Generic (PLEG): container finished" podID="0a00e7bb-8155-4318-9959-2d488a6d1d79" containerID="cb70437fa4472d7989ed879179edc2019fd17cbb0118046152fc4fe16726f173" exitCode=0 Jul 09 04:07:32.109059 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:32.108982 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dbd6b8ff4-52f2p" Jul 09 04:07:32.109059 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:32.109031 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dbd6b8ff4-52f2p" event={"ID":"0a00e7bb-8155-4318-9959-2d488a6d1d79","Type":"ContainerDied","Data":"cb70437fa4472d7989ed879179edc2019fd17cbb0118046152fc4fe16726f173"} Jul 09 04:07:32.109371 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:32.109069 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dbd6b8ff4-52f2p" event={"ID":"0a00e7bb-8155-4318-9959-2d488a6d1d79","Type":"ContainerDied","Data":"16b632a52f0a2509bd83ab4272a00d920dc4ce8358c2bf421e9ec1de6f75a051"} Jul 09 04:07:32.109371 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:32.109087 2571 scope.go:117] "RemoveContainer" containerID="cb70437fa4472d7989ed879179edc2019fd17cbb0118046152fc4fe16726f173" Jul 09 04:07:32.118755 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:32.118741 2571 scope.go:117] "RemoveContainer" containerID="cb70437fa4472d7989ed879179edc2019fd17cbb0118046152fc4fe16726f173" Jul 09 04:07:32.119010 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:07:32.118994 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb70437fa4472d7989ed879179edc2019fd17cbb0118046152fc4fe16726f173\": container with ID starting with cb70437fa4472d7989ed879179edc2019fd17cbb0118046152fc4fe16726f173 not found: ID does not exist" containerID="cb70437fa4472d7989ed879179edc2019fd17cbb0118046152fc4fe16726f173" Jul 09 04:07:32.119080 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:32.119018 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb70437fa4472d7989ed879179edc2019fd17cbb0118046152fc4fe16726f173"} err="failed to get container status \"cb70437fa4472d7989ed879179edc2019fd17cbb0118046152fc4fe16726f173\": rpc error: code = NotFound desc = could not find container \"cb70437fa4472d7989ed879179edc2019fd17cbb0118046152fc4fe16726f173\": container with ID starting with cb70437fa4472d7989ed879179edc2019fd17cbb0118046152fc4fe16726f173 not found: ID does not exist" Jul 09 04:07:32.130268 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:32.130248 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dbd6b8ff4-52f2p"] Jul 09 04:07:32.133898 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:32.133850 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dbd6b8ff4-52f2p"] Jul 09 04:07:33.626323 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:33.626291 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a00e7bb-8155-4318-9959-2d488a6d1d79" path="/var/lib/kubelet/pods/0a00e7bb-8155-4318-9959-2d488a6d1d79/volumes" Jul 09 04:07:59.452927 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:07:59.452838 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:08:09.147719 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:08:09.147683 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:08:20.943215 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:08:20.943180 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:08:29.443633 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:08:29.443603 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:08:39.045570 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:08:39.045532 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:09:06.341007 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:09:06.340973 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:09:20.054889 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:09:20.054846 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:09:53.934088 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:09:53.934052 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:10:01.648606 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:10:01.648573 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:10:12.934730 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:10:12.934696 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:10:20.243792 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:10:20.243755 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:10:31.654708 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:10:31.654677 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:10:42.842419 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:10:42.842376 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:10:56.138058 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:10:56.138021 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:11:05.229215 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:11:05.229126 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:11:05.600303 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:11:05.600279 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:11:05.601488 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:11:05.601467 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:11:29.733113 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:11:29.733083 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:11:42.430023 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:11:42.429988 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:12:05.832027 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:12:05.831984 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:12:21.236461 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:12:21.236425 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:12:32.633560 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:12:32.633475 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:12:42.945022 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:12:42.944990 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:12:53.142255 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:12:53.142222 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:13:05.132134 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:13:05.132106 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:15:00.542461 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:15:00.542428 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:15:12.946712 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:15:12.946681 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:15:24.134669 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:15:24.134634 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:15:36.543161 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:15:36.543079 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:15:48.242664 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:15:48.237809 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:15:58.042454 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:15:58.042422 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:16:05.635531 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:16:05.635501 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:16:05.635955 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:16:05.635899 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:16:07.941354 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:16:07.941320 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:16:17.837117 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:16:17.837087 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:16:29.340349 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:16:29.340316 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:16:37.855925 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:16:37.855887 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:16:49.037226 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:16:49.037188 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:17:13.442013 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:17:13.441930 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:17:21.441626 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:17:21.441589 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:17:34.243870 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:17:34.243826 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:17:44.139766 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:17:44.139734 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:18:02.037221 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:18:02.037188 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:18:16.338375 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:18:16.338338 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:18:26.230497 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:18:26.230465 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:18:40.149032 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:18:40.148947 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:18:49.741585 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:18:49.741544 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:19:00.744797 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:19:00.744754 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:19:17.735478 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:19:17.735450 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:19:27.734385 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:19:27.734348 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:19:34.939036 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:19:34.939003 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:19:45.230080 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:19:45.230049 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:19:55.535625 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:19:55.535584 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:20:07.343914 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:20:07.343821 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:20:20.736716 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:20:20.736683 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:20:33.040788 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:20:33.040749 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:20:45.343705 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:20:45.343566 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:20:54.528934 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:20:54.528897 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:21:04.954450 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:04.954419 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:21:05.669197 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:05.669169 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:21:05.672159 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:05.672135 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:21:21.742788 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:21.742743 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:21:33.334288 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:33.334249 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:21:44.945657 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:44.945625 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:21:49.764824 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:49.764784 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr"] Jul 09 04:21:49.765235 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:49.765198 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a00e7bb-8155-4318-9959-2d488a6d1d79" containerName="authorino" Jul 09 04:21:49.765235 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:49.765213 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a00e7bb-8155-4318-9959-2d488a6d1d79" containerName="authorino" Jul 09 04:21:49.765309 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:49.765275 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a00e7bb-8155-4318-9959-2d488a6d1d79" containerName="authorino" Jul 09 04:21:49.768376 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:49.768359 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" Jul 09 04:21:49.773125 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:49.773098 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 04:21:49.773314 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:49.773290 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-znghr\"" Jul 09 04:21:49.774038 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:49.773926 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 04:21:49.774038 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:49.773926 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-64447b93-serving-cert\"" Jul 09 04:21:49.806915 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:49.806886 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr"] Jul 09 04:21:49.905248 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:49.905215 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdj67\" (UniqueName: \"kubernetes.io/projected/69c45e79-4c9f-492d-9ab8-61edc0cd2428-kube-api-access-bdj67\") pod \"maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr\" (UID: \"69c45e79-4c9f-492d-9ab8-61edc0cd2428\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" Jul 09 04:21:49.905416 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:49.905264 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69c45e79-4c9f-492d-9ab8-61edc0cd2428-maas-api-tls\") pod \"maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr\" (UID: \"69c45e79-4c9f-492d-9ab8-61edc0cd2428\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" Jul 09 04:21:49.905416 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:49.905294 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c45e79-4c9f-492d-9ab8-61edc0cd2428-openshift-service-ca\") pod \"maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr\" (UID: \"69c45e79-4c9f-492d-9ab8-61edc0cd2428\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" Jul 09 04:21:50.006239 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:50.006196 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdj67\" (UniqueName: \"kubernetes.io/projected/69c45e79-4c9f-492d-9ab8-61edc0cd2428-kube-api-access-bdj67\") pod \"maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr\" (UID: \"69c45e79-4c9f-492d-9ab8-61edc0cd2428\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" Jul 09 04:21:50.006423 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:50.006253 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69c45e79-4c9f-492d-9ab8-61edc0cd2428-maas-api-tls\") pod \"maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr\" (UID: \"69c45e79-4c9f-492d-9ab8-61edc0cd2428\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" Jul 09 04:21:50.006423 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:50.006286 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c45e79-4c9f-492d-9ab8-61edc0cd2428-openshift-service-ca\") pod \"maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr\" (UID: \"69c45e79-4c9f-492d-9ab8-61edc0cd2428\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" Jul 09 04:21:50.007085 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:50.007065 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c45e79-4c9f-492d-9ab8-61edc0cd2428-openshift-service-ca\") pod \"maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr\" (UID: \"69c45e79-4c9f-492d-9ab8-61edc0cd2428\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" Jul 09 04:21:50.008777 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:50.008753 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69c45e79-4c9f-492d-9ab8-61edc0cd2428-maas-api-tls\") pod \"maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr\" (UID: \"69c45e79-4c9f-492d-9ab8-61edc0cd2428\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" Jul 09 04:21:50.018356 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:50.018302 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdj67\" (UniqueName: \"kubernetes.io/projected/69c45e79-4c9f-492d-9ab8-61edc0cd2428-kube-api-access-bdj67\") pod \"maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr\" (UID: \"69c45e79-4c9f-492d-9ab8-61edc0cd2428\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" Jul 09 04:21:50.080661 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:50.080631 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" Jul 09 04:21:50.235313 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:50.235290 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr"] Jul 09 04:21:50.237456 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:21:50.237431 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c45e79_4c9f_492d_9ab8_61edc0cd2428.slice/crio-609a12035e270b54675c7e043f80d11aed6f358cf3737ed2df09587d602ae990 WatchSource:0}: Error finding container 609a12035e270b54675c7e043f80d11aed6f358cf3737ed2df09587d602ae990: Status 404 returned error can't find the container with id 609a12035e270b54675c7e043f80d11aed6f358cf3737ed2df09587d602ae990 Jul 09 04:21:50.238774 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:50.238758 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 04:21:50.603024 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:50.602987 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" event={"ID":"69c45e79-4c9f-492d-9ab8-61edc0cd2428","Type":"ContainerStarted","Data":"609a12035e270b54675c7e043f80d11aed6f358cf3737ed2df09587d602ae990"} Jul 09 04:21:52.616803 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:52.616767 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" event={"ID":"69c45e79-4c9f-492d-9ab8-61edc0cd2428","Type":"ContainerStarted","Data":"ac636596e003a28edf0051931b361bac23854eba9c162081ca4ff5113910d1f0"} Jul 09 04:21:52.617222 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:52.616913 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" Jul 09 04:21:52.638556 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:52.638501 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" podStartSLOduration=2.045126973 podStartE2EDuration="3.638482412s" totalImagesPullingTime="1.593355439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:21:52.637638414 +0000 UTC m=+1547.496509645" watchObservedRunningTime="2026-07-09 04:21:52.638482412 +0000 UTC m=+1547.497353515" Jul 09 04:21:54.361531 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:54.361497 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr"] Jul 09 04:21:54.628180 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:54.628075 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" podUID="69c45e79-4c9f-492d-9ab8-61edc0cd2428" containerName="maas-api" containerID="cri-o://ac636596e003a28edf0051931b361bac23854eba9c162081ca4ff5113910d1f0" gracePeriod=30 Jul 09 04:21:54.877804 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:54.877777 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" Jul 09 04:21:55.053833 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.053739 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdj67\" (UniqueName: \"kubernetes.io/projected/69c45e79-4c9f-492d-9ab8-61edc0cd2428-kube-api-access-bdj67\") pod \"69c45e79-4c9f-492d-9ab8-61edc0cd2428\" (UID: \"69c45e79-4c9f-492d-9ab8-61edc0cd2428\") " Jul 09 04:21:55.053833 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.053783 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69c45e79-4c9f-492d-9ab8-61edc0cd2428-maas-api-tls\") pod \"69c45e79-4c9f-492d-9ab8-61edc0cd2428\" (UID: \"69c45e79-4c9f-492d-9ab8-61edc0cd2428\") " Jul 09 04:21:55.053833 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.053835 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c45e79-4c9f-492d-9ab8-61edc0cd2428-openshift-service-ca\") pod \"69c45e79-4c9f-492d-9ab8-61edc0cd2428\" (UID: \"69c45e79-4c9f-492d-9ab8-61edc0cd2428\") " Jul 09 04:21:55.054231 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.054209 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c45e79-4c9f-492d-9ab8-61edc0cd2428-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "69c45e79-4c9f-492d-9ab8-61edc0cd2428" (UID: "69c45e79-4c9f-492d-9ab8-61edc0cd2428"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:21:55.056036 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.056017 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69c45e79-4c9f-492d-9ab8-61edc0cd2428-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "69c45e79-4c9f-492d-9ab8-61edc0cd2428" (UID: "69c45e79-4c9f-492d-9ab8-61edc0cd2428"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:21:55.056259 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.056230 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c45e79-4c9f-492d-9ab8-61edc0cd2428-kube-api-access-bdj67" (OuterVolumeSpecName: "kube-api-access-bdj67") pod "69c45e79-4c9f-492d-9ab8-61edc0cd2428" (UID: "69c45e79-4c9f-492d-9ab8-61edc0cd2428"). InnerVolumeSpecName "kube-api-access-bdj67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:21:55.155210 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.155171 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69c45e79-4c9f-492d-9ab8-61edc0cd2428-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:21:55.155210 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.155204 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c45e79-4c9f-492d-9ab8-61edc0cd2428-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:21:55.155210 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.155216 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdj67\" (UniqueName: \"kubernetes.io/projected/69c45e79-4c9f-492d-9ab8-61edc0cd2428-kube-api-access-bdj67\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:21:55.638121 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.638083 2571 generic.go:356] "Generic (PLEG): container finished" podID="69c45e79-4c9f-492d-9ab8-61edc0cd2428" containerID="ac636596e003a28edf0051931b361bac23854eba9c162081ca4ff5113910d1f0" exitCode=0 Jul 09 04:21:55.638525 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.638207 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" Jul 09 04:21:55.642355 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.642328 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" event={"ID":"69c45e79-4c9f-492d-9ab8-61edc0cd2428","Type":"ContainerDied","Data":"ac636596e003a28edf0051931b361bac23854eba9c162081ca4ff5113910d1f0"} Jul 09 04:21:55.642521 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.642377 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr" event={"ID":"69c45e79-4c9f-492d-9ab8-61edc0cd2428","Type":"ContainerDied","Data":"609a12035e270b54675c7e043f80d11aed6f358cf3737ed2df09587d602ae990"} Jul 09 04:21:55.642521 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.642402 2571 scope.go:117] "RemoveContainer" containerID="ac636596e003a28edf0051931b361bac23854eba9c162081ca4ff5113910d1f0" Jul 09 04:21:55.652138 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.652117 2571 scope.go:117] "RemoveContainer" containerID="ac636596e003a28edf0051931b361bac23854eba9c162081ca4ff5113910d1f0" Jul 09 04:21:55.652390 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:21:55.652373 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac636596e003a28edf0051931b361bac23854eba9c162081ca4ff5113910d1f0\": container with ID starting with ac636596e003a28edf0051931b361bac23854eba9c162081ca4ff5113910d1f0 not found: ID does not exist" containerID="ac636596e003a28edf0051931b361bac23854eba9c162081ca4ff5113910d1f0" Jul 09 04:21:55.652438 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.652398 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac636596e003a28edf0051931b361bac23854eba9c162081ca4ff5113910d1f0"} err="failed to get container status \"ac636596e003a28edf0051931b361bac23854eba9c162081ca4ff5113910d1f0\": rpc error: code = NotFound desc = could not find container \"ac636596e003a28edf0051931b361bac23854eba9c162081ca4ff5113910d1f0\": container with ID starting with ac636596e003a28edf0051931b361bac23854eba9c162081ca4ff5113910d1f0 not found: ID does not exist" Jul 09 04:21:55.666668 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.666636 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr"] Jul 09 04:21:55.672409 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:55.672373 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64447b93-9558cdd6d-kfwhr"] Jul 09 04:21:57.626606 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:21:57.626567 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69c45e79-4c9f-492d-9ab8-61edc0cd2428" path="/var/lib/kubelet/pods/69c45e79-4c9f-492d-9ab8-61edc0cd2428/volumes" Jul 09 04:23:23.249985 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:23:23.249903 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:23:31.051790 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:23:31.051753 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:23:39.945283 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:23:39.945249 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:23:50.460942 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:23:50.460911 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:23:57.609366 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:23:57.609316 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:24:23.307962 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.307934 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct"] Jul 09 04:24:23.308434 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.308284 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69c45e79-4c9f-492d-9ab8-61edc0cd2428" containerName="maas-api" Jul 09 04:24:23.308434 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.308295 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="69c45e79-4c9f-492d-9ab8-61edc0cd2428" containerName="maas-api" Jul 09 04:24:23.308434 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.308373 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="69c45e79-4c9f-492d-9ab8-61edc0cd2428" containerName="maas-api" Jul 09 04:24:23.311456 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.311437 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" Jul 09 04:24:23.316405 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.316380 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 04:24:23.316839 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.316820 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-znghr\"" Jul 09 04:24:23.317728 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.317709 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1986c679-serving-cert\"" Jul 09 04:24:23.317898 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.317709 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 04:24:23.324267 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.324248 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct"] Jul 09 04:24:23.435659 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.435629 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/385637aa-55a0-4661-89b1-ffb12cf0989a-maas-api-tls\") pod \"maas-api-e2e-mt-1986c679-85c8c4f56-cmcct\" (UID: \"385637aa-55a0-4661-89b1-ffb12cf0989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" Jul 09 04:24:23.435804 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.435675 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7scjl\" (UniqueName: \"kubernetes.io/projected/385637aa-55a0-4661-89b1-ffb12cf0989a-kube-api-access-7scjl\") pod \"maas-api-e2e-mt-1986c679-85c8c4f56-cmcct\" (UID: \"385637aa-55a0-4661-89b1-ffb12cf0989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" Jul 09 04:24:23.435804 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.435734 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/385637aa-55a0-4661-89b1-ffb12cf0989a-openshift-service-ca\") pod \"maas-api-e2e-mt-1986c679-85c8c4f56-cmcct\" (UID: \"385637aa-55a0-4661-89b1-ffb12cf0989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" Jul 09 04:24:23.536643 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.536604 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/385637aa-55a0-4661-89b1-ffb12cf0989a-openshift-service-ca\") pod \"maas-api-e2e-mt-1986c679-85c8c4f56-cmcct\" (UID: \"385637aa-55a0-4661-89b1-ffb12cf0989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" Jul 09 04:24:23.536783 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.536659 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/385637aa-55a0-4661-89b1-ffb12cf0989a-maas-api-tls\") pod \"maas-api-e2e-mt-1986c679-85c8c4f56-cmcct\" (UID: \"385637aa-55a0-4661-89b1-ffb12cf0989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" Jul 09 04:24:23.536783 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.536693 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7scjl\" (UniqueName: \"kubernetes.io/projected/385637aa-55a0-4661-89b1-ffb12cf0989a-kube-api-access-7scjl\") pod \"maas-api-e2e-mt-1986c679-85c8c4f56-cmcct\" (UID: \"385637aa-55a0-4661-89b1-ffb12cf0989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" Jul 09 04:24:23.536887 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:24:23.536828 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-serving-cert: secret "maas-api-e2e-mt-1986c679-serving-cert" not found Jul 09 04:24:23.536944 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:24:23.536931 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/385637aa-55a0-4661-89b1-ffb12cf0989a-maas-api-tls podName:385637aa-55a0-4661-89b1-ffb12cf0989a nodeName:}" failed. No retries permitted until 2026-07-09 04:24:24.036906443 +0000 UTC m=+1698.895777525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/385637aa-55a0-4661-89b1-ffb12cf0989a-maas-api-tls") pod "maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" (UID: "385637aa-55a0-4661-89b1-ffb12cf0989a") : secret "maas-api-e2e-mt-1986c679-serving-cert" not found Jul 09 04:24:23.537335 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.537317 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/385637aa-55a0-4661-89b1-ffb12cf0989a-openshift-service-ca\") pod \"maas-api-e2e-mt-1986c679-85c8c4f56-cmcct\" (UID: \"385637aa-55a0-4661-89b1-ffb12cf0989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" Jul 09 04:24:23.544819 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:23.544793 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7scjl\" (UniqueName: \"kubernetes.io/projected/385637aa-55a0-4661-89b1-ffb12cf0989a-kube-api-access-7scjl\") pod \"maas-api-e2e-mt-1986c679-85c8c4f56-cmcct\" (UID: \"385637aa-55a0-4661-89b1-ffb12cf0989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" Jul 09 04:24:24.041048 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:24.041012 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/385637aa-55a0-4661-89b1-ffb12cf0989a-maas-api-tls\") pod \"maas-api-e2e-mt-1986c679-85c8c4f56-cmcct\" (UID: \"385637aa-55a0-4661-89b1-ffb12cf0989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" Jul 09 04:24:24.043271 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:24.043252 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/385637aa-55a0-4661-89b1-ffb12cf0989a-maas-api-tls\") pod \"maas-api-e2e-mt-1986c679-85c8c4f56-cmcct\" (UID: \"385637aa-55a0-4661-89b1-ffb12cf0989a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" Jul 09 04:24:24.229091 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:24.229058 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" Jul 09 04:24:24.360205 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:24.360180 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct"] Jul 09 04:24:24.361537 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:24:24.361512 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385637aa_55a0_4661_89b1_ffb12cf0989a.slice/crio-7ec175cf9354d1cbc5ab2ce3629aa8b3944afeb199a56dd3b032f0ca1b48ea57 WatchSource:0}: Error finding container 7ec175cf9354d1cbc5ab2ce3629aa8b3944afeb199a56dd3b032f0ca1b48ea57: Status 404 returned error can't find the container with id 7ec175cf9354d1cbc5ab2ce3629aa8b3944afeb199a56dd3b032f0ca1b48ea57 Jul 09 04:24:24.397913 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:24.397886 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" event={"ID":"385637aa-55a0-4661-89b1-ffb12cf0989a","Type":"ContainerStarted","Data":"7ec175cf9354d1cbc5ab2ce3629aa8b3944afeb199a56dd3b032f0ca1b48ea57"} Jul 09 04:24:25.404508 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:25.404477 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" event={"ID":"385637aa-55a0-4661-89b1-ffb12cf0989a","Type":"ContainerStarted","Data":"82ec437812f1caf33e087a8d08573e5a49fbe5e97164ddb7c6e925ccb277056c"} Jul 09 04:24:25.404931 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:25.404563 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" Jul 09 04:24:25.421568 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:25.421523 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" podStartSLOduration=2.109340184 podStartE2EDuration="2.421509909s" totalImagesPullingTime="312.169725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:24:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:24:25.419295572 +0000 UTC m=+1700.278166672" watchObservedRunningTime="2026-07-09 04:24:25.421509909 +0000 UTC m=+1700.280381009" Jul 09 04:24:26.637964 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:26.637930 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:24:31.417482 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:31.417450 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" Jul 09 04:24:36.839497 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:36.839468 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp"] Jul 09 04:24:36.843017 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:36.842999 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:36.845713 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:36.845692 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1986c679\"/\"default-dockercfg-2jsgp\"" Jul 09 04:24:36.845818 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:36.845710 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1986c679\"/\"kube-root-ca.crt\"" Jul 09 04:24:36.846327 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:36.846310 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1986c679\"/\"openshift-service-ca.crt\"" Jul 09 04:24:36.846610 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:36.846598 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1986c679\"/\"e2e-lifecycle-model-1986c679-kserve-self-signed-certs\"" Jul 09 04:24:36.853915 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:36.853893 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp"] Jul 09 04:24:36.944358 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:36.944334 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-home\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:36.944571 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:36.944369 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4def9fd8-07cb-4990-8772-a4cff2decf96-tls-certs\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:36.944571 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:36.944390 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-model-cache\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:36.944571 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:36.944420 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g9qk\" (UniqueName: \"kubernetes.io/projected/4def9fd8-07cb-4990-8772-a4cff2decf96-kube-api-access-6g9qk\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:36.944571 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:36.944498 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-tmp-dir\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:36.944571 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:36.944546 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-dshm\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:37.045656 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.045630 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4def9fd8-07cb-4990-8772-a4cff2decf96-tls-certs\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:37.045807 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.045664 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-model-cache\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:37.045807 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.045692 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g9qk\" (UniqueName: \"kubernetes.io/projected/4def9fd8-07cb-4990-8772-a4cff2decf96-kube-api-access-6g9qk\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:37.045807 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.045717 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-tmp-dir\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:37.045807 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.045748 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-dshm\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:37.045807 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.045782 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-home\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:37.046134 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.046114 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-model-cache\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:37.046196 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.046139 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-home\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:37.046236 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.046194 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-tmp-dir\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:37.047895 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.047848 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-dshm\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:37.048160 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.048141 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4def9fd8-07cb-4990-8772-a4cff2decf96-tls-certs\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:37.054430 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.054409 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g9qk\" (UniqueName: \"kubernetes.io/projected/4def9fd8-07cb-4990-8772-a4cff2decf96-kube-api-access-6g9qk\") pod \"e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:37.152137 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.152116 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:37.287690 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.287659 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp"] Jul 09 04:24:37.288629 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:24:37.288606 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4def9fd8_07cb_4990_8772_a4cff2decf96.slice/crio-b7275b3bb6e6b495680caef6b945480da97e63242d53db23ecb0d872b0140644 WatchSource:0}: Error finding container b7275b3bb6e6b495680caef6b945480da97e63242d53db23ecb0d872b0140644: Status 404 returned error can't find the container with id b7275b3bb6e6b495680caef6b945480da97e63242d53db23ecb0d872b0140644 Jul 09 04:24:37.468755 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.468679 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" event={"ID":"4def9fd8-07cb-4990-8772-a4cff2decf96","Type":"ContainerStarted","Data":"b2a576e62fca9c76d8d4fcf836be9496c644240bfd6566ea1f98e91c0ef0665f"} Jul 09 04:24:37.468755 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.468716 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" event={"ID":"4def9fd8-07cb-4990-8772-a4cff2decf96","Type":"ContainerStarted","Data":"b7275b3bb6e6b495680caef6b945480da97e63242d53db23ecb0d872b0140644"} Jul 09 04:24:37.488139 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:37.488090 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" podStartSLOduration=1.488070043 podStartE2EDuration="1.488070043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:24:37.487894065 +0000 UTC m=+1712.346765164" watchObservedRunningTime="2026-07-09 04:24:37.488070043 +0000 UTC m=+1712.346941143" Jul 09 04:24:40.736541 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:40.736503 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:24:47.153223 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:47.153189 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:47.153644 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:47.153237 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:47.165425 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:47.165390 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:47.534740 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:47.534670 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:24:52.996840 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:52.996803 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:24:55.190665 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.190634 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct"] Jul 09 04:24:55.191150 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.190820 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" podUID="385637aa-55a0-4661-89b1-ffb12cf0989a" containerName="maas-api" containerID="cri-o://82ec437812f1caf33e087a8d08573e5a49fbe5e97164ddb7c6e925ccb277056c" gracePeriod=30 Jul 09 04:24:55.440167 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.440140 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" Jul 09 04:24:55.497559 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.497494 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7scjl\" (UniqueName: \"kubernetes.io/projected/385637aa-55a0-4661-89b1-ffb12cf0989a-kube-api-access-7scjl\") pod \"385637aa-55a0-4661-89b1-ffb12cf0989a\" (UID: \"385637aa-55a0-4661-89b1-ffb12cf0989a\") " Jul 09 04:24:55.497559 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.497555 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/385637aa-55a0-4661-89b1-ffb12cf0989a-maas-api-tls\") pod \"385637aa-55a0-4661-89b1-ffb12cf0989a\" (UID: \"385637aa-55a0-4661-89b1-ffb12cf0989a\") " Jul 09 04:24:55.497722 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.497596 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/385637aa-55a0-4661-89b1-ffb12cf0989a-openshift-service-ca\") pod \"385637aa-55a0-4661-89b1-ffb12cf0989a\" (UID: \"385637aa-55a0-4661-89b1-ffb12cf0989a\") " Jul 09 04:24:55.498010 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.497983 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/385637aa-55a0-4661-89b1-ffb12cf0989a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "385637aa-55a0-4661-89b1-ffb12cf0989a" (UID: "385637aa-55a0-4661-89b1-ffb12cf0989a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:24:55.499406 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.499378 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385637aa-55a0-4661-89b1-ffb12cf0989a-kube-api-access-7scjl" (OuterVolumeSpecName: "kube-api-access-7scjl") pod "385637aa-55a0-4661-89b1-ffb12cf0989a" (UID: "385637aa-55a0-4661-89b1-ffb12cf0989a"). InnerVolumeSpecName "kube-api-access-7scjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:24:55.499498 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.499453 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/385637aa-55a0-4661-89b1-ffb12cf0989a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "385637aa-55a0-4661-89b1-ffb12cf0989a" (UID: "385637aa-55a0-4661-89b1-ffb12cf0989a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:24:55.568183 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.568158 2571 generic.go:356] "Generic (PLEG): container finished" podID="385637aa-55a0-4661-89b1-ffb12cf0989a" containerID="82ec437812f1caf33e087a8d08573e5a49fbe5e97164ddb7c6e925ccb277056c" exitCode=0 Jul 09 04:24:55.568285 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.568201 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" Jul 09 04:24:55.568285 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.568242 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" event={"ID":"385637aa-55a0-4661-89b1-ffb12cf0989a","Type":"ContainerDied","Data":"82ec437812f1caf33e087a8d08573e5a49fbe5e97164ddb7c6e925ccb277056c"} Jul 09 04:24:55.568359 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.568288 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct" event={"ID":"385637aa-55a0-4661-89b1-ffb12cf0989a","Type":"ContainerDied","Data":"7ec175cf9354d1cbc5ab2ce3629aa8b3944afeb199a56dd3b032f0ca1b48ea57"} Jul 09 04:24:55.568359 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.568310 2571 scope.go:117] "RemoveContainer" containerID="82ec437812f1caf33e087a8d08573e5a49fbe5e97164ddb7c6e925ccb277056c" Jul 09 04:24:55.577001 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.576987 2571 scope.go:117] "RemoveContainer" containerID="82ec437812f1caf33e087a8d08573e5a49fbe5e97164ddb7c6e925ccb277056c" Jul 09 04:24:55.577235 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:24:55.577218 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ec437812f1caf33e087a8d08573e5a49fbe5e97164ddb7c6e925ccb277056c\": container with ID starting with 82ec437812f1caf33e087a8d08573e5a49fbe5e97164ddb7c6e925ccb277056c not found: ID does not exist" containerID="82ec437812f1caf33e087a8d08573e5a49fbe5e97164ddb7c6e925ccb277056c" Jul 09 04:24:55.577292 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.577242 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ec437812f1caf33e087a8d08573e5a49fbe5e97164ddb7c6e925ccb277056c"} err="failed to get container status \"82ec437812f1caf33e087a8d08573e5a49fbe5e97164ddb7c6e925ccb277056c\": rpc error: code = NotFound desc = could not find container \"82ec437812f1caf33e087a8d08573e5a49fbe5e97164ddb7c6e925ccb277056c\": container with ID starting with 82ec437812f1caf33e087a8d08573e5a49fbe5e97164ddb7c6e925ccb277056c not found: ID does not exist" Jul 09 04:24:55.588876 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.588844 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct"] Jul 09 04:24:55.594738 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.594716 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1986c679-85c8c4f56-cmcct"] Jul 09 04:24:55.598727 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.598711 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/385637aa-55a0-4661-89b1-ffb12cf0989a-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:24:55.598788 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.598730 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/385637aa-55a0-4661-89b1-ffb12cf0989a-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:24:55.598788 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.598739 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7scjl\" (UniqueName: \"kubernetes.io/projected/385637aa-55a0-4661-89b1-ffb12cf0989a-kube-api-access-7scjl\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:24:55.627403 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:24:55.627382 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="385637aa-55a0-4661-89b1-ffb12cf0989a" path="/var/lib/kubelet/pods/385637aa-55a0-4661-89b1-ffb12cf0989a/volumes" Jul 09 04:25:05.881802 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:05.881771 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp"] Jul 09 04:25:06.844558 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:06.844530 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:25:20.900533 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:20.900476 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" podUID="4def9fd8-07cb-4990-8772-a4cff2decf96" containerName="main" containerID="cri-o://b2a576e62fca9c76d8d4fcf836be9496c644240bfd6566ea1f98e91c0ef0665f" gracePeriod=45 Jul 09 04:25:21.144189 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.144168 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:25:21.283910 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.283825 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4def9fd8-07cb-4990-8772-a4cff2decf96-tls-certs\") pod \"4def9fd8-07cb-4990-8772-a4cff2decf96\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " Jul 09 04:25:21.283910 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.283886 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g9qk\" (UniqueName: \"kubernetes.io/projected/4def9fd8-07cb-4990-8772-a4cff2decf96-kube-api-access-6g9qk\") pod \"4def9fd8-07cb-4990-8772-a4cff2decf96\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " Jul 09 04:25:21.284093 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.283927 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-model-cache\") pod \"4def9fd8-07cb-4990-8772-a4cff2decf96\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " Jul 09 04:25:21.284093 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.283984 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-home\") pod \"4def9fd8-07cb-4990-8772-a4cff2decf96\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " Jul 09 04:25:21.284093 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.284006 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-tmp-dir\") pod \"4def9fd8-07cb-4990-8772-a4cff2decf96\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " Jul 09 04:25:21.284093 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.284049 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-dshm\") pod \"4def9fd8-07cb-4990-8772-a4cff2decf96\" (UID: \"4def9fd8-07cb-4990-8772-a4cff2decf96\") " Jul 09 04:25:21.284294 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.284268 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-home" (OuterVolumeSpecName: "home") pod "4def9fd8-07cb-4990-8772-a4cff2decf96" (UID: "4def9fd8-07cb-4990-8772-a4cff2decf96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:25:21.284294 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.284281 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4def9fd8-07cb-4990-8772-a4cff2decf96" (UID: "4def9fd8-07cb-4990-8772-a4cff2decf96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:25:21.284386 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.284295 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-model-cache" (OuterVolumeSpecName: "model-cache") pod "4def9fd8-07cb-4990-8772-a4cff2decf96" (UID: "4def9fd8-07cb-4990-8772-a4cff2decf96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:25:21.285997 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.285968 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-dshm" (OuterVolumeSpecName: "dshm") pod "4def9fd8-07cb-4990-8772-a4cff2decf96" (UID: "4def9fd8-07cb-4990-8772-a4cff2decf96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:25:21.285997 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.285977 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4def9fd8-07cb-4990-8772-a4cff2decf96-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4def9fd8-07cb-4990-8772-a4cff2decf96" (UID: "4def9fd8-07cb-4990-8772-a4cff2decf96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:25:21.286133 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.285995 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4def9fd8-07cb-4990-8772-a4cff2decf96-kube-api-access-6g9qk" (OuterVolumeSpecName: "kube-api-access-6g9qk") pod "4def9fd8-07cb-4990-8772-a4cff2decf96" (UID: "4def9fd8-07cb-4990-8772-a4cff2decf96"). InnerVolumeSpecName "kube-api-access-6g9qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:25:21.385028 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.385006 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4def9fd8-07cb-4990-8772-a4cff2decf96-tls-certs\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:25:21.385028 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.385025 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g9qk\" (UniqueName: \"kubernetes.io/projected/4def9fd8-07cb-4990-8772-a4cff2decf96-kube-api-access-6g9qk\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:25:21.385161 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.385035 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-model-cache\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:25:21.385161 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.385043 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-home\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:25:21.385161 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.385050 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-tmp-dir\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:25:21.385161 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.385058 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4def9fd8-07cb-4990-8772-a4cff2decf96-dshm\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:25:21.702491 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.702464 2571 generic.go:356] "Generic (PLEG): container finished" podID="4def9fd8-07cb-4990-8772-a4cff2decf96" containerID="b2a576e62fca9c76d8d4fcf836be9496c644240bfd6566ea1f98e91c0ef0665f" exitCode=0 Jul 09 04:25:21.702595 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.702514 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" event={"ID":"4def9fd8-07cb-4990-8772-a4cff2decf96","Type":"ContainerDied","Data":"b2a576e62fca9c76d8d4fcf836be9496c644240bfd6566ea1f98e91c0ef0665f"} Jul 09 04:25:21.702595 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.702519 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" Jul 09 04:25:21.702595 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.702546 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp" event={"ID":"4def9fd8-07cb-4990-8772-a4cff2decf96","Type":"ContainerDied","Data":"b7275b3bb6e6b495680caef6b945480da97e63242d53db23ecb0d872b0140644"} Jul 09 04:25:21.702595 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.702563 2571 scope.go:117] "RemoveContainer" containerID="b2a576e62fca9c76d8d4fcf836be9496c644240bfd6566ea1f98e91c0ef0665f" Jul 09 04:25:21.711296 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.711278 2571 scope.go:117] "RemoveContainer" containerID="b2a576e62fca9c76d8d4fcf836be9496c644240bfd6566ea1f98e91c0ef0665f" Jul 09 04:25:21.711539 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:25:21.711522 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a576e62fca9c76d8d4fcf836be9496c644240bfd6566ea1f98e91c0ef0665f\": container with ID starting with b2a576e62fca9c76d8d4fcf836be9496c644240bfd6566ea1f98e91c0ef0665f not found: ID does not exist" containerID="b2a576e62fca9c76d8d4fcf836be9496c644240bfd6566ea1f98e91c0ef0665f" Jul 09 04:25:21.711594 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.711545 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a576e62fca9c76d8d4fcf836be9496c644240bfd6566ea1f98e91c0ef0665f"} err="failed to get container status \"b2a576e62fca9c76d8d4fcf836be9496c644240bfd6566ea1f98e91c0ef0665f\": rpc error: code = NotFound desc = could not find container \"b2a576e62fca9c76d8d4fcf836be9496c644240bfd6566ea1f98e91c0ef0665f\": container with ID starting with b2a576e62fca9c76d8d4fcf836be9496c644240bfd6566ea1f98e91c0ef0665f not found: ID does not exist" Jul 09 04:25:21.718893 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.718873 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp"] Jul 09 04:25:21.723083 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:21.723065 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-1986c679/e2e-lifecycle-model-1986c679-kserve-955588c75-gv6vp"] Jul 09 04:25:23.626029 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:23.625994 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4def9fd8-07cb-4990-8772-a4cff2decf96" path="/var/lib/kubelet/pods/4def9fd8-07cb-4990-8772-a4cff2decf96/volumes" Jul 09 04:25:45.344336 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:45.344305 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:25:58.144981 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:25:58.144901 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:26:05.703374 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:05.703344 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:26:05.706723 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:05.706700 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:26:12.129504 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:12.129466 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:26:59.875537 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:59.875507 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28"] Jul 09 04:26:59.876003 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:59.875836 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4def9fd8-07cb-4990-8772-a4cff2decf96" containerName="main" Jul 09 04:26:59.876003 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:59.875848 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4def9fd8-07cb-4990-8772-a4cff2decf96" containerName="main" Jul 09 04:26:59.876003 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:59.875871 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="385637aa-55a0-4661-89b1-ffb12cf0989a" containerName="maas-api" Jul 09 04:26:59.876003 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:59.875877 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="385637aa-55a0-4661-89b1-ffb12cf0989a" containerName="maas-api" Jul 09 04:26:59.876003 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:59.875989 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4def9fd8-07cb-4990-8772-a4cff2decf96" containerName="main" Jul 09 04:26:59.876287 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:59.876007 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="385637aa-55a0-4661-89b1-ffb12cf0989a" containerName="maas-api" Jul 09 04:26:59.879114 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:59.879099 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" Jul 09 04:26:59.881775 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:59.881752 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 04:26:59.881898 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:59.881762 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-znghr\"" Jul 09 04:26:59.881898 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:59.881873 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6fdfb1-serving-cert\"" Jul 09 04:26:59.882343 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:59.882330 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 04:26:59.889777 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:59.889756 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28"] Jul 09 04:26:59.987321 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:59.987298 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72jlk\" (UniqueName: \"kubernetes.io/projected/796f49d6-2e8d-4f09-a382-03de92c9696c-kube-api-access-72jlk\") pod \"maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28\" (UID: \"796f49d6-2e8d-4f09-a382-03de92c9696c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" Jul 09 04:26:59.987445 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:59.987333 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/796f49d6-2e8d-4f09-a382-03de92c9696c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28\" (UID: \"796f49d6-2e8d-4f09-a382-03de92c9696c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" Jul 09 04:26:59.987445 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:26:59.987360 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/796f49d6-2e8d-4f09-a382-03de92c9696c-maas-api-tls\") pod \"maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28\" (UID: \"796f49d6-2e8d-4f09-a382-03de92c9696c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" Jul 09 04:27:00.088371 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:00.088348 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72jlk\" (UniqueName: \"kubernetes.io/projected/796f49d6-2e8d-4f09-a382-03de92c9696c-kube-api-access-72jlk\") pod \"maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28\" (UID: \"796f49d6-2e8d-4f09-a382-03de92c9696c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" Jul 09 04:27:00.088492 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:00.088380 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/796f49d6-2e8d-4f09-a382-03de92c9696c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28\" (UID: \"796f49d6-2e8d-4f09-a382-03de92c9696c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" Jul 09 04:27:00.088492 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:00.088407 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/796f49d6-2e8d-4f09-a382-03de92c9696c-maas-api-tls\") pod \"maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28\" (UID: \"796f49d6-2e8d-4f09-a382-03de92c9696c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" Jul 09 04:27:00.088604 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:27:00.088521 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-serving-cert: secret "maas-api-e2e-inf-b-6fdfb1-serving-cert" not found Jul 09 04:27:00.088604 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:27:00.088581 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/796f49d6-2e8d-4f09-a382-03de92c9696c-maas-api-tls podName:796f49d6-2e8d-4f09-a382-03de92c9696c nodeName:}" failed. No retries permitted until 2026-07-09 04:27:00.588564393 +0000 UTC m=+1855.447435471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/796f49d6-2e8d-4f09-a382-03de92c9696c-maas-api-tls") pod "maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" (UID: "796f49d6-2e8d-4f09-a382-03de92c9696c") : secret "maas-api-e2e-inf-b-6fdfb1-serving-cert" not found Jul 09 04:27:00.089069 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:00.089052 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/796f49d6-2e8d-4f09-a382-03de92c9696c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28\" (UID: \"796f49d6-2e8d-4f09-a382-03de92c9696c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" Jul 09 04:27:00.097171 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:00.097151 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:27:00.103166 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:00.103149 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72jlk\" (UniqueName: \"kubernetes.io/projected/796f49d6-2e8d-4f09-a382-03de92c9696c-kube-api-access-72jlk\") pod \"maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28\" (UID: \"796f49d6-2e8d-4f09-a382-03de92c9696c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" Jul 09 04:27:00.592032 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:00.592003 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/796f49d6-2e8d-4f09-a382-03de92c9696c-maas-api-tls\") pod \"maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28\" (UID: \"796f49d6-2e8d-4f09-a382-03de92c9696c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" Jul 09 04:27:00.594217 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:00.594201 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/796f49d6-2e8d-4f09-a382-03de92c9696c-maas-api-tls\") pod \"maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28\" (UID: \"796f49d6-2e8d-4f09-a382-03de92c9696c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" Jul 09 04:27:00.791029 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:00.791003 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" Jul 09 04:27:00.921217 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:00.921190 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28"] Jul 09 04:27:00.926150 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:00.926126 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 04:27:01.191774 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:01.191689 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" event={"ID":"796f49d6-2e8d-4f09-a382-03de92c9696c","Type":"ContainerStarted","Data":"6740dff473c2978db736dc9eb44b26267421db72e35e4074e03f073042e1f376"} Jul 09 04:27:02.198041 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:02.198010 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" event={"ID":"796f49d6-2e8d-4f09-a382-03de92c9696c","Type":"ContainerStarted","Data":"baa956fb85fbe6c7c5485075f74caefd7d0612bdbaa3bab2d0585ff05b774143"} Jul 09 04:27:02.198402 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:02.198125 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" Jul 09 04:27:02.214184 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:02.214140 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" podStartSLOduration=2.632111524 podStartE2EDuration="3.214127164s" totalImagesPullingTime="582.01564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:26:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:27:02.212134137 +0000 UTC m=+1857.071005235" watchObservedRunningTime="2026-07-09 04:27:02.214127164 +0000 UTC m=+1857.072998262" Jul 09 04:27:08.208870 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:08.208838 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" Jul 09 04:27:11.239744 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.239711 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:27:11.629255 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.629226 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x"] Jul 09 04:27:11.632584 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.632562 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.634818 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.634794 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e6ba0c\"/\"kube-root-ca.crt\"" Jul 09 04:27:11.635648 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.635618 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e6ba0c\"/\"default-dockercfg-dvqst\"" Jul 09 04:27:11.635648 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.635644 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e6ba0c\"/\"openshift-service-ca.crt\"" Jul 09 04:27:11.635838 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.635618 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e6ba0c\"/\"test-model-e6ba0c-kserve-self-signed-certs\"" Jul 09 04:27:11.642578 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.642560 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x"] Jul 09 04:27:11.770927 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.770906 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47045579-6b3a-4b85-b1df-f3ffa2ea0527-tls-certs\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.771032 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.770961 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxj2k\" (UniqueName: \"kubernetes.io/projected/47045579-6b3a-4b85-b1df-f3ffa2ea0527-kube-api-access-lxj2k\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.771032 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.771018 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-home\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.771111 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.771039 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-dshm\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.771111 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.771078 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-model-cache\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.771111 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.771094 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-tmp-dir\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.871692 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.871671 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxj2k\" (UniqueName: \"kubernetes.io/projected/47045579-6b3a-4b85-b1df-f3ffa2ea0527-kube-api-access-lxj2k\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.871791 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.871714 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-home\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.871791 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.871734 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-dshm\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.871791 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.871781 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-model-cache\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.871947 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.871804 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-tmp-dir\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.871947 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.871830 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47045579-6b3a-4b85-b1df-f3ffa2ea0527-tls-certs\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.872125 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.872105 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-home\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.872188 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.872162 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-tmp-dir\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.872188 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.872182 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-model-cache\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.873972 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.873949 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-dshm\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.874137 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.874123 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47045579-6b3a-4b85-b1df-f3ffa2ea0527-tls-certs\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.879130 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.879110 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxj2k\" (UniqueName: \"kubernetes.io/projected/47045579-6b3a-4b85-b1df-f3ffa2ea0527-kube-api-access-lxj2k\") pod \"test-model-e6ba0c-kserve-8597c4c56c-cbs5x\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:11.944382 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:11.944328 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:12.076591 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:12.071576 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x"] Jul 09 04:27:12.250790 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:12.250714 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" event={"ID":"47045579-6b3a-4b85-b1df-f3ffa2ea0527","Type":"ContainerStarted","Data":"c3f4b3af9dc3bb6aa30c84c3600e1250ba3dddc452bcfe3ccd8a6047c4e8a72e"} Jul 09 04:27:12.250790 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:12.250752 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" event={"ID":"47045579-6b3a-4b85-b1df-f3ffa2ea0527","Type":"ContainerStarted","Data":"5ff7c770d9bc5f0648d147831e3650f54002c88541755f3ae103926edb458423"} Jul 09 04:27:12.267899 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:12.267836 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" podStartSLOduration=1.267822449 podStartE2EDuration="1.267822449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:27:12.266116385 +0000 UTC m=+1867.124987487" watchObservedRunningTime="2026-07-09 04:27:12.267822449 +0000 UTC m=+1867.126693574" Jul 09 04:27:21.843872 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:21.843828 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:27:21.944639 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:21.944612 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:21.944799 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:21.944681 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:21.956935 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:21.956913 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:27:22.314496 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:27:22.314436 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:28:03.938571 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:03.938546 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv"] Jul 09 04:28:03.941947 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:03.941923 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:03.945071 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:03.945048 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6fdfb1\"/\"test-model-6fdfb1-kserve-self-signed-certs\"" Jul 09 04:28:03.945202 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:03.945071 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6fdfb1\"/\"default-dockercfg-8gjfr\"" Jul 09 04:28:03.945202 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:03.945071 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6fdfb1\"/\"kube-root-ca.crt\"" Jul 09 04:28:03.945202 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:03.945114 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6fdfb1\"/\"openshift-service-ca.crt\"" Jul 09 04:28:03.950661 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:03.950641 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv"] Jul 09 04:28:04.057910 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.057883 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-tmp-dir\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.058059 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.057929 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-model-cache\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.058059 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.058002 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-dshm\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.058059 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.058025 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb531c4b-d1c6-4d41-bcef-a284f600fe19-tls-certs\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.058188 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.058071 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h86pk\" (UniqueName: \"kubernetes.io/projected/eb531c4b-d1c6-4d41-bcef-a284f600fe19-kube-api-access-h86pk\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.058188 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.058146 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-home\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.159178 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.159152 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-model-cache\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.159309 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.159191 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-dshm\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.159309 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.159206 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb531c4b-d1c6-4d41-bcef-a284f600fe19-tls-certs\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.159419 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.159328 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h86pk\" (UniqueName: \"kubernetes.io/projected/eb531c4b-d1c6-4d41-bcef-a284f600fe19-kube-api-access-h86pk\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.159419 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.159388 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-home\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.159419 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.159416 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-tmp-dir\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.159631 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.159612 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-model-cache\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.159699 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.159691 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-home\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.159812 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.159791 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-tmp-dir\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.161310 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.161289 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-dshm\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.161498 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.161480 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb531c4b-d1c6-4d41-bcef-a284f600fe19-tls-certs\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.169429 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.169408 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h86pk\" (UniqueName: \"kubernetes.io/projected/eb531c4b-d1c6-4d41-bcef-a284f600fe19-kube-api-access-h86pk\") pod \"test-model-6fdfb1-kserve-65bcf687b7-md5bv\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.251879 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.251805 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:04.376817 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.376785 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv"] Jul 09 04:28:04.378094 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:28:04.378067 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb531c4b_d1c6_4d41_bcef_a284f600fe19.slice/crio-7732c99adb13129bc90fedf53e12a47a55278d6b5ec1f502cefbfa64ffab8c44 WatchSource:0}: Error finding container 7732c99adb13129bc90fedf53e12a47a55278d6b5ec1f502cefbfa64ffab8c44: Status 404 returned error can't find the container with id 7732c99adb13129bc90fedf53e12a47a55278d6b5ec1f502cefbfa64ffab8c44 Jul 09 04:28:04.517634 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.517572 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" event={"ID":"eb531c4b-d1c6-4d41-bcef-a284f600fe19","Type":"ContainerStarted","Data":"2ed89baa34f70ab53bd30be07851fd6f452ad6f8b18b1bc7e68603d22ee8f100"} Jul 09 04:28:04.517634 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.517609 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" event={"ID":"eb531c4b-d1c6-4d41-bcef-a284f600fe19","Type":"ContainerStarted","Data":"7732c99adb13129bc90fedf53e12a47a55278d6b5ec1f502cefbfa64ffab8c44"} Jul 09 04:28:04.534254 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:04.534214 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" podStartSLOduration=1.534199424 podStartE2EDuration="1.534199424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:28:04.531915386 +0000 UTC m=+1919.390786484" watchObservedRunningTime="2026-07-09 04:28:04.534199424 +0000 UTC m=+1919.393070524" Jul 09 04:28:09.028266 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:09.028238 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:28:14.252670 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:14.252636 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:14.252670 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:14.252677 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:14.264630 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:14.264609 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:14.577127 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:14.577059 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:22.535672 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:22.535638 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:28:22.979520 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:22.979494 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x"] Jul 09 04:28:24.013450 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:24.013418 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv"] Jul 09 04:28:34.938651 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:34.938618 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:28:37.999124 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:37.999067 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" podUID="47045579-6b3a-4b85-b1df-f3ffa2ea0527" containerName="main" containerID="cri-o://c3f4b3af9dc3bb6aa30c84c3600e1250ba3dddc452bcfe3ccd8a6047c4e8a72e" gracePeriod=45 Jul 09 04:28:38.240424 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.240397 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:28:38.313827 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.313710 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-dshm\") pod \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " Jul 09 04:28:38.313956 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.313833 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47045579-6b3a-4b85-b1df-f3ffa2ea0527-tls-certs\") pod \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " Jul 09 04:28:38.313956 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.313887 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-model-cache\") pod \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " Jul 09 04:28:38.313956 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.313931 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-home\") pod \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " Jul 09 04:28:38.313956 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.313952 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-tmp-dir\") pod \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " Jul 09 04:28:38.314157 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.313991 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxj2k\" (UniqueName: \"kubernetes.io/projected/47045579-6b3a-4b85-b1df-f3ffa2ea0527-kube-api-access-lxj2k\") pod \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\" (UID: \"47045579-6b3a-4b85-b1df-f3ffa2ea0527\") " Jul 09 04:28:38.314212 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.314188 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-home" (OuterVolumeSpecName: "home") pod "47045579-6b3a-4b85-b1df-f3ffa2ea0527" (UID: "47045579-6b3a-4b85-b1df-f3ffa2ea0527"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:28:38.314260 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.314203 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-model-cache" (OuterVolumeSpecName: "model-cache") pod "47045579-6b3a-4b85-b1df-f3ffa2ea0527" (UID: "47045579-6b3a-4b85-b1df-f3ffa2ea0527"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:28:38.314260 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.314217 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47045579-6b3a-4b85-b1df-f3ffa2ea0527" (UID: "47045579-6b3a-4b85-b1df-f3ffa2ea0527"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:28:38.314358 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.314321 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-model-cache\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:28:38.314358 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.314339 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-home\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:28:38.314437 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.314373 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-tmp-dir\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:28:38.315777 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.315753 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-dshm" (OuterVolumeSpecName: "dshm") pod "47045579-6b3a-4b85-b1df-f3ffa2ea0527" (UID: "47045579-6b3a-4b85-b1df-f3ffa2ea0527"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:28:38.315899 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.315834 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47045579-6b3a-4b85-b1df-f3ffa2ea0527-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47045579-6b3a-4b85-b1df-f3ffa2ea0527" (UID: "47045579-6b3a-4b85-b1df-f3ffa2ea0527"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:28:38.315963 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.315948 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47045579-6b3a-4b85-b1df-f3ffa2ea0527-kube-api-access-lxj2k" (OuterVolumeSpecName: "kube-api-access-lxj2k") pod "47045579-6b3a-4b85-b1df-f3ffa2ea0527" (UID: "47045579-6b3a-4b85-b1df-f3ffa2ea0527"). InnerVolumeSpecName "kube-api-access-lxj2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:28:38.415438 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.415419 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxj2k\" (UniqueName: \"kubernetes.io/projected/47045579-6b3a-4b85-b1df-f3ffa2ea0527-kube-api-access-lxj2k\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:28:38.415528 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.415439 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47045579-6b3a-4b85-b1df-f3ffa2ea0527-dshm\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:28:38.415528 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.415450 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47045579-6b3a-4b85-b1df-f3ffa2ea0527-tls-certs\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:28:38.688145 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.688113 2571 generic.go:356] "Generic (PLEG): container finished" podID="47045579-6b3a-4b85-b1df-f3ffa2ea0527" containerID="c3f4b3af9dc3bb6aa30c84c3600e1250ba3dddc452bcfe3ccd8a6047c4e8a72e" exitCode=0 Jul 09 04:28:38.688283 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.688154 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" Jul 09 04:28:38.688283 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.688200 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" event={"ID":"47045579-6b3a-4b85-b1df-f3ffa2ea0527","Type":"ContainerDied","Data":"c3f4b3af9dc3bb6aa30c84c3600e1250ba3dddc452bcfe3ccd8a6047c4e8a72e"} Jul 09 04:28:38.688283 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.688241 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x" event={"ID":"47045579-6b3a-4b85-b1df-f3ffa2ea0527","Type":"ContainerDied","Data":"5ff7c770d9bc5f0648d147831e3650f54002c88541755f3ae103926edb458423"} Jul 09 04:28:38.688283 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.688258 2571 scope.go:117] "RemoveContainer" containerID="c3f4b3af9dc3bb6aa30c84c3600e1250ba3dddc452bcfe3ccd8a6047c4e8a72e" Jul 09 04:28:38.696892 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.696876 2571 scope.go:117] "RemoveContainer" containerID="c3f4b3af9dc3bb6aa30c84c3600e1250ba3dddc452bcfe3ccd8a6047c4e8a72e" Jul 09 04:28:38.697150 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:28:38.697132 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3f4b3af9dc3bb6aa30c84c3600e1250ba3dddc452bcfe3ccd8a6047c4e8a72e\": container with ID starting with c3f4b3af9dc3bb6aa30c84c3600e1250ba3dddc452bcfe3ccd8a6047c4e8a72e not found: ID does not exist" containerID="c3f4b3af9dc3bb6aa30c84c3600e1250ba3dddc452bcfe3ccd8a6047c4e8a72e" Jul 09 04:28:38.697221 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.697157 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f4b3af9dc3bb6aa30c84c3600e1250ba3dddc452bcfe3ccd8a6047c4e8a72e"} err="failed to get container status \"c3f4b3af9dc3bb6aa30c84c3600e1250ba3dddc452bcfe3ccd8a6047c4e8a72e\": rpc error: code = NotFound desc = could not find container \"c3f4b3af9dc3bb6aa30c84c3600e1250ba3dddc452bcfe3ccd8a6047c4e8a72e\": container with ID starting with c3f4b3af9dc3bb6aa30c84c3600e1250ba3dddc452bcfe3ccd8a6047c4e8a72e not found: ID does not exist" Jul 09 04:28:38.708570 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.708550 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x"] Jul 09 04:28:38.712953 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:38.712934 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e6ba0c/test-model-e6ba0c-kserve-8597c4c56c-cbs5x"] Jul 09 04:28:39.032677 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.032608 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" podUID="eb531c4b-d1c6-4d41-bcef-a284f600fe19" containerName="main" containerID="cri-o://2ed89baa34f70ab53bd30be07851fd6f452ad6f8b18b1bc7e68603d22ee8f100" gracePeriod=45 Jul 09 04:28:39.277847 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.277824 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:39.324258 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.324207 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-dshm\") pod \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " Jul 09 04:28:39.324258 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.324248 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-model-cache\") pod \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " Jul 09 04:28:39.324429 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.324285 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h86pk\" (UniqueName: \"kubernetes.io/projected/eb531c4b-d1c6-4d41-bcef-a284f600fe19-kube-api-access-h86pk\") pod \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " Jul 09 04:28:39.324429 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.324308 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb531c4b-d1c6-4d41-bcef-a284f600fe19-tls-certs\") pod \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " Jul 09 04:28:39.324429 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.324345 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-home\") pod \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " Jul 09 04:28:39.324585 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.324488 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-tmp-dir\") pod \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\" (UID: \"eb531c4b-d1c6-4d41-bcef-a284f600fe19\") " Jul 09 04:28:39.324585 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.324556 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb531c4b-d1c6-4d41-bcef-a284f600fe19" (UID: "eb531c4b-d1c6-4d41-bcef-a284f600fe19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:28:39.324695 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.324591 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-home" (OuterVolumeSpecName: "home") pod "eb531c4b-d1c6-4d41-bcef-a284f600fe19" (UID: "eb531c4b-d1c6-4d41-bcef-a284f600fe19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:28:39.324775 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.324754 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb531c4b-d1c6-4d41-bcef-a284f600fe19" (UID: "eb531c4b-d1c6-4d41-bcef-a284f600fe19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:28:39.324836 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.324798 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-model-cache\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:28:39.324836 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.324811 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-home\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:28:39.326380 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.326360 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-dshm" (OuterVolumeSpecName: "dshm") pod "eb531c4b-d1c6-4d41-bcef-a284f600fe19" (UID: "eb531c4b-d1c6-4d41-bcef-a284f600fe19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:28:39.326736 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.326716 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb531c4b-d1c6-4d41-bcef-a284f600fe19-kube-api-access-h86pk" (OuterVolumeSpecName: "kube-api-access-h86pk") pod "eb531c4b-d1c6-4d41-bcef-a284f600fe19" (UID: "eb531c4b-d1c6-4d41-bcef-a284f600fe19"). InnerVolumeSpecName "kube-api-access-h86pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:28:39.326808 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.326718 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb531c4b-d1c6-4d41-bcef-a284f600fe19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb531c4b-d1c6-4d41-bcef-a284f600fe19" (UID: "eb531c4b-d1c6-4d41-bcef-a284f600fe19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:28:39.425755 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.425733 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-tmp-dir\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:28:39.425755 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.425752 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb531c4b-d1c6-4d41-bcef-a284f600fe19-dshm\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:28:39.425902 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.425762 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h86pk\" (UniqueName: \"kubernetes.io/projected/eb531c4b-d1c6-4d41-bcef-a284f600fe19-kube-api-access-h86pk\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:28:39.425902 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.425772 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb531c4b-d1c6-4d41-bcef-a284f600fe19-tls-certs\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:28:39.625919 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.625897 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47045579-6b3a-4b85-b1df-f3ffa2ea0527" path="/var/lib/kubelet/pods/47045579-6b3a-4b85-b1df-f3ffa2ea0527/volumes" Jul 09 04:28:39.694085 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.694060 2571 generic.go:356] "Generic (PLEG): container finished" podID="eb531c4b-d1c6-4d41-bcef-a284f600fe19" containerID="2ed89baa34f70ab53bd30be07851fd6f452ad6f8b18b1bc7e68603d22ee8f100" exitCode=0 Jul 09 04:28:39.694233 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.694098 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" Jul 09 04:28:39.694233 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.694142 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" event={"ID":"eb531c4b-d1c6-4d41-bcef-a284f600fe19","Type":"ContainerDied","Data":"2ed89baa34f70ab53bd30be07851fd6f452ad6f8b18b1bc7e68603d22ee8f100"} Jul 09 04:28:39.694233 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.694179 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv" event={"ID":"eb531c4b-d1c6-4d41-bcef-a284f600fe19","Type":"ContainerDied","Data":"7732c99adb13129bc90fedf53e12a47a55278d6b5ec1f502cefbfa64ffab8c44"} Jul 09 04:28:39.694233 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.694202 2571 scope.go:117] "RemoveContainer" containerID="2ed89baa34f70ab53bd30be07851fd6f452ad6f8b18b1bc7e68603d22ee8f100" Jul 09 04:28:39.702734 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.702719 2571 scope.go:117] "RemoveContainer" containerID="2ed89baa34f70ab53bd30be07851fd6f452ad6f8b18b1bc7e68603d22ee8f100" Jul 09 04:28:39.703021 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:28:39.703004 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed89baa34f70ab53bd30be07851fd6f452ad6f8b18b1bc7e68603d22ee8f100\": container with ID starting with 2ed89baa34f70ab53bd30be07851fd6f452ad6f8b18b1bc7e68603d22ee8f100 not found: ID does not exist" containerID="2ed89baa34f70ab53bd30be07851fd6f452ad6f8b18b1bc7e68603d22ee8f100" Jul 09 04:28:39.703100 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.703030 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed89baa34f70ab53bd30be07851fd6f452ad6f8b18b1bc7e68603d22ee8f100"} err="failed to get container status \"2ed89baa34f70ab53bd30be07851fd6f452ad6f8b18b1bc7e68603d22ee8f100\": rpc error: code = NotFound desc = could not find container \"2ed89baa34f70ab53bd30be07851fd6f452ad6f8b18b1bc7e68603d22ee8f100\": container with ID starting with 2ed89baa34f70ab53bd30be07851fd6f452ad6f8b18b1bc7e68603d22ee8f100 not found: ID does not exist" Jul 09 04:28:39.709862 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.709829 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv"] Jul 09 04:28:39.713469 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:39.713449 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6fdfb1/test-model-6fdfb1-kserve-65bcf687b7-md5bv"] Jul 09 04:28:41.626036 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:28:41.626007 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb531c4b-d1c6-4d41-bcef-a284f600fe19" path="/var/lib/kubelet/pods/eb531c4b-d1c6-4d41-bcef-a284f600fe19/volumes" Jul 09 04:29:07.666206 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:07.666171 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28"] Jul 09 04:29:07.666675 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:07.666393 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" podUID="796f49d6-2e8d-4f09-a382-03de92c9696c" containerName="maas-api" containerID="cri-o://baa956fb85fbe6c7c5485075f74caefd7d0612bdbaa3bab2d0585ff05b774143" gracePeriod=30 Jul 09 04:29:07.835865 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:07.835819 2571 generic.go:356] "Generic (PLEG): container finished" podID="796f49d6-2e8d-4f09-a382-03de92c9696c" containerID="baa956fb85fbe6c7c5485075f74caefd7d0612bdbaa3bab2d0585ff05b774143" exitCode=0 Jul 09 04:29:07.835988 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:07.835905 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" event={"ID":"796f49d6-2e8d-4f09-a382-03de92c9696c","Type":"ContainerDied","Data":"baa956fb85fbe6c7c5485075f74caefd7d0612bdbaa3bab2d0585ff05b774143"} Jul 09 04:29:07.909202 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:07.909176 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" Jul 09 04:29:08.028394 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:08.028333 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/796f49d6-2e8d-4f09-a382-03de92c9696c-maas-api-tls\") pod \"796f49d6-2e8d-4f09-a382-03de92c9696c\" (UID: \"796f49d6-2e8d-4f09-a382-03de92c9696c\") " Jul 09 04:29:08.028485 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:08.028414 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/796f49d6-2e8d-4f09-a382-03de92c9696c-openshift-service-ca\") pod \"796f49d6-2e8d-4f09-a382-03de92c9696c\" (UID: \"796f49d6-2e8d-4f09-a382-03de92c9696c\") " Jul 09 04:29:08.028485 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:08.028443 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72jlk\" (UniqueName: \"kubernetes.io/projected/796f49d6-2e8d-4f09-a382-03de92c9696c-kube-api-access-72jlk\") pod \"796f49d6-2e8d-4f09-a382-03de92c9696c\" (UID: \"796f49d6-2e8d-4f09-a382-03de92c9696c\") " Jul 09 04:29:08.028764 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:08.028744 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/796f49d6-2e8d-4f09-a382-03de92c9696c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "796f49d6-2e8d-4f09-a382-03de92c9696c" (UID: "796f49d6-2e8d-4f09-a382-03de92c9696c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:29:08.030415 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:08.030385 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796f49d6-2e8d-4f09-a382-03de92c9696c-kube-api-access-72jlk" (OuterVolumeSpecName: "kube-api-access-72jlk") pod "796f49d6-2e8d-4f09-a382-03de92c9696c" (UID: "796f49d6-2e8d-4f09-a382-03de92c9696c"). InnerVolumeSpecName "kube-api-access-72jlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:29:08.030415 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:08.030405 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/796f49d6-2e8d-4f09-a382-03de92c9696c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "796f49d6-2e8d-4f09-a382-03de92c9696c" (UID: "796f49d6-2e8d-4f09-a382-03de92c9696c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:29:08.129415 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:08.129393 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/796f49d6-2e8d-4f09-a382-03de92c9696c-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:29:08.129415 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:08.129412 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/796f49d6-2e8d-4f09-a382-03de92c9696c-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:29:08.129538 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:08.129421 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72jlk\" (UniqueName: \"kubernetes.io/projected/796f49d6-2e8d-4f09-a382-03de92c9696c-kube-api-access-72jlk\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:29:08.842456 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:08.842423 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" event={"ID":"796f49d6-2e8d-4f09-a382-03de92c9696c","Type":"ContainerDied","Data":"6740dff473c2978db736dc9eb44b26267421db72e35e4074e03f073042e1f376"} Jul 09 04:29:08.842941 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:08.842494 2571 scope.go:117] "RemoveContainer" containerID="baa956fb85fbe6c7c5485075f74caefd7d0612bdbaa3bab2d0585ff05b774143" Jul 09 04:29:08.842941 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:08.842433 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28" Jul 09 04:29:08.864499 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:08.864475 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28"] Jul 09 04:29:08.874081 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:08.870688 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6fdfb1-75896685f7-wcs28"] Jul 09 04:29:09.625673 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:09.625635 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="796f49d6-2e8d-4f09-a382-03de92c9696c" path="/var/lib/kubelet/pods/796f49d6-2e8d-4f09-a382-03de92c9696c/volumes" Jul 09 04:29:10.936027 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:10.935993 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:29:17.439798 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.439769 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4"] Jul 09 04:29:17.440164 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.440113 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="796f49d6-2e8d-4f09-a382-03de92c9696c" containerName="maas-api" Jul 09 04:29:17.440164 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.440126 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="796f49d6-2e8d-4f09-a382-03de92c9696c" containerName="maas-api" Jul 09 04:29:17.440164 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.440137 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb531c4b-d1c6-4d41-bcef-a284f600fe19" containerName="main" Jul 09 04:29:17.440164 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.440143 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb531c4b-d1c6-4d41-bcef-a284f600fe19" containerName="main" Jul 09 04:29:17.440164 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.440163 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47045579-6b3a-4b85-b1df-f3ffa2ea0527" containerName="main" Jul 09 04:29:17.440325 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.440168 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="47045579-6b3a-4b85-b1df-f3ffa2ea0527" containerName="main" Jul 09 04:29:17.440325 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.440219 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb531c4b-d1c6-4d41-bcef-a284f600fe19" containerName="main" Jul 09 04:29:17.440325 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.440227 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="796f49d6-2e8d-4f09-a382-03de92c9696c" containerName="maas-api" Jul 09 04:29:17.440325 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.440236 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="47045579-6b3a-4b85-b1df-f3ffa2ea0527" containerName="main" Jul 09 04:29:17.444428 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.444404 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" Jul 09 04:29:17.447402 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.447376 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2cb3ca-serving-cert\"" Jul 09 04:29:17.447529 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.447461 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 04:29:17.447666 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.447652 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-znghr\"" Jul 09 04:29:17.448177 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.448158 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 04:29:17.457951 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.457931 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4"] Jul 09 04:29:17.598640 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.598611 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7txc\" (UniqueName: \"kubernetes.io/projected/33088a46-fcc7-48f1-90c4-43fc1abedf99-kube-api-access-t7txc\") pod \"maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4\" (UID: \"33088a46-fcc7-48f1-90c4-43fc1abedf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" Jul 09 04:29:17.598740 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.598671 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33088a46-fcc7-48f1-90c4-43fc1abedf99-maas-api-tls\") pod \"maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4\" (UID: \"33088a46-fcc7-48f1-90c4-43fc1abedf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" Jul 09 04:29:17.598740 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.598696 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33088a46-fcc7-48f1-90c4-43fc1abedf99-openshift-service-ca\") pod \"maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4\" (UID: \"33088a46-fcc7-48f1-90c4-43fc1abedf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" Jul 09 04:29:17.699881 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.699811 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33088a46-fcc7-48f1-90c4-43fc1abedf99-maas-api-tls\") pod \"maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4\" (UID: \"33088a46-fcc7-48f1-90c4-43fc1abedf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" Jul 09 04:29:17.699881 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.699838 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33088a46-fcc7-48f1-90c4-43fc1abedf99-openshift-service-ca\") pod \"maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4\" (UID: \"33088a46-fcc7-48f1-90c4-43fc1abedf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" Jul 09 04:29:17.700006 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.699907 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7txc\" (UniqueName: \"kubernetes.io/projected/33088a46-fcc7-48f1-90c4-43fc1abedf99-kube-api-access-t7txc\") pod \"maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4\" (UID: \"33088a46-fcc7-48f1-90c4-43fc1abedf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" Jul 09 04:29:17.700538 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.700515 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33088a46-fcc7-48f1-90c4-43fc1abedf99-openshift-service-ca\") pod \"maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4\" (UID: \"33088a46-fcc7-48f1-90c4-43fc1abedf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" Jul 09 04:29:17.702133 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.702114 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33088a46-fcc7-48f1-90c4-43fc1abedf99-maas-api-tls\") pod \"maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4\" (UID: \"33088a46-fcc7-48f1-90c4-43fc1abedf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" Jul 09 04:29:17.709590 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.709568 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7txc\" (UniqueName: \"kubernetes.io/projected/33088a46-fcc7-48f1-90c4-43fc1abedf99-kube-api-access-t7txc\") pod \"maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4\" (UID: \"33088a46-fcc7-48f1-90c4-43fc1abedf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" Jul 09 04:29:17.754795 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.754773 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" Jul 09 04:29:17.904675 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:17.904646 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4"] Jul 09 04:29:17.908274 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:29:17.908247 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33088a46_fcc7_48f1_90c4_43fc1abedf99.slice/crio-b583eba26eb6e0fa484bb49259f0b8edd5c67578dea9bb283a41623b29f8fb57 WatchSource:0}: Error finding container b583eba26eb6e0fa484bb49259f0b8edd5c67578dea9bb283a41623b29f8fb57: Status 404 returned error can't find the container with id b583eba26eb6e0fa484bb49259f0b8edd5c67578dea9bb283a41623b29f8fb57 Jul 09 04:29:18.894438 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:18.894403 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" event={"ID":"33088a46-fcc7-48f1-90c4-43fc1abedf99","Type":"ContainerStarted","Data":"86b988ef72e3a9764c44409ad126a5027edbc8ecd983fd901892774fc7a854a4"} Jul 09 04:29:18.894793 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:18.894449 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" event={"ID":"33088a46-fcc7-48f1-90c4-43fc1abedf99","Type":"ContainerStarted","Data":"b583eba26eb6e0fa484bb49259f0b8edd5c67578dea9bb283a41623b29f8fb57"} Jul 09 04:29:18.894793 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:18.894490 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" Jul 09 04:29:18.914128 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:18.914083 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" podStartSLOduration=1.5178395839999999 podStartE2EDuration="1.914071239s" totalImagesPullingTime="396.231655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:29:18.913298488 +0000 UTC m=+1993.772169579" watchObservedRunningTime="2026-07-09 04:29:18.914071239 +0000 UTC m=+1993.772942339" Jul 09 04:29:24.651214 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:24.651176 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:29:24.905115 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:24.905046 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" Jul 09 04:29:25.488503 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:25.488474 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg"] Jul 09 04:29:25.492133 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:25.492110 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" Jul 09 04:29:25.495066 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:25.495041 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d61384-serving-cert\"" Jul 09 04:29:25.506410 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:25.506386 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg"] Jul 09 04:29:25.659747 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:25.659725 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gggt\" (UniqueName: \"kubernetes.io/projected/b269836e-e387-454b-9e53-b73aa51b6ea6-kube-api-access-7gggt\") pod \"maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg\" (UID: \"b269836e-e387-454b-9e53-b73aa51b6ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" Jul 09 04:29:25.660135 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:25.659758 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b269836e-e387-454b-9e53-b73aa51b6ea6-maas-api-tls\") pod \"maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg\" (UID: \"b269836e-e387-454b-9e53-b73aa51b6ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" Jul 09 04:29:25.660135 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:25.659878 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b269836e-e387-454b-9e53-b73aa51b6ea6-openshift-service-ca\") pod \"maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg\" (UID: \"b269836e-e387-454b-9e53-b73aa51b6ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" Jul 09 04:29:25.760253 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:25.760182 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b269836e-e387-454b-9e53-b73aa51b6ea6-openshift-service-ca\") pod \"maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg\" (UID: \"b269836e-e387-454b-9e53-b73aa51b6ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" Jul 09 04:29:25.760253 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:25.760240 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gggt\" (UniqueName: \"kubernetes.io/projected/b269836e-e387-454b-9e53-b73aa51b6ea6-kube-api-access-7gggt\") pod \"maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg\" (UID: \"b269836e-e387-454b-9e53-b73aa51b6ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" Jul 09 04:29:25.760435 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:25.760275 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b269836e-e387-454b-9e53-b73aa51b6ea6-maas-api-tls\") pod \"maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg\" (UID: \"b269836e-e387-454b-9e53-b73aa51b6ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" Jul 09 04:29:25.760781 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:25.760759 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b269836e-e387-454b-9e53-b73aa51b6ea6-openshift-service-ca\") pod \"maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg\" (UID: \"b269836e-e387-454b-9e53-b73aa51b6ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" Jul 09 04:29:25.762542 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:25.762524 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b269836e-e387-454b-9e53-b73aa51b6ea6-maas-api-tls\") pod \"maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg\" (UID: \"b269836e-e387-454b-9e53-b73aa51b6ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" Jul 09 04:29:25.769432 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:25.769405 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gggt\" (UniqueName: \"kubernetes.io/projected/b269836e-e387-454b-9e53-b73aa51b6ea6-kube-api-access-7gggt\") pod \"maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg\" (UID: \"b269836e-e387-454b-9e53-b73aa51b6ea6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" Jul 09 04:29:25.804567 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:25.804549 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" Jul 09 04:29:25.941000 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:25.940977 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg"] Jul 09 04:29:25.942636 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:29:25.942612 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb269836e_e387_454b_9e53_b73aa51b6ea6.slice/crio-f7c6df8e7d24c73fd59a8eda0d8e3ee6df76666e5b917b11616f58f116c38cc7 WatchSource:0}: Error finding container f7c6df8e7d24c73fd59a8eda0d8e3ee6df76666e5b917b11616f58f116c38cc7: Status 404 returned error can't find the container with id f7c6df8e7d24c73fd59a8eda0d8e3ee6df76666e5b917b11616f58f116c38cc7 Jul 09 04:29:26.935078 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:26.935041 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" event={"ID":"b269836e-e387-454b-9e53-b73aa51b6ea6","Type":"ContainerStarted","Data":"e92d1d54e193a72609dce209bd3d5a51a47a2f8ae8347001106c6ae414060c42"} Jul 09 04:29:26.935483 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:26.935135 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" Jul 09 04:29:26.935483 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:26.935162 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" event={"ID":"b269836e-e387-454b-9e53-b73aa51b6ea6","Type":"ContainerStarted","Data":"f7c6df8e7d24c73fd59a8eda0d8e3ee6df76666e5b917b11616f58f116c38cc7"} Jul 09 04:29:26.967132 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:26.967090 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" podStartSLOduration=1.510413421 podStartE2EDuration="1.967075756s" totalImagesPullingTime="456.662335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:29:26.965829441 +0000 UTC m=+2001.824700540" watchObservedRunningTime="2026-07-09 04:29:26.967075756 +0000 UTC m=+2001.825946854" Jul 09 04:29:32.945573 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:32.945548 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" Jul 09 04:29:38.869159 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:38.869129 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4"] Jul 09 04:29:38.869646 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:38.869327 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" podUID="33088a46-fcc7-48f1-90c4-43fc1abedf99" containerName="maas-api" containerID="cri-o://86b988ef72e3a9764c44409ad126a5027edbc8ecd983fd901892774fc7a854a4" gracePeriod=30 Jul 09 04:29:38.997808 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:38.997773 2571 generic.go:356] "Generic (PLEG): container finished" podID="33088a46-fcc7-48f1-90c4-43fc1abedf99" containerID="86b988ef72e3a9764c44409ad126a5027edbc8ecd983fd901892774fc7a854a4" exitCode=0 Jul 09 04:29:38.997940 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:38.997842 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" event={"ID":"33088a46-fcc7-48f1-90c4-43fc1abedf99","Type":"ContainerDied","Data":"86b988ef72e3a9764c44409ad126a5027edbc8ecd983fd901892774fc7a854a4"} Jul 09 04:29:39.116211 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:39.116191 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" Jul 09 04:29:39.163646 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:39.163581 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33088a46-fcc7-48f1-90c4-43fc1abedf99-maas-api-tls\") pod \"33088a46-fcc7-48f1-90c4-43fc1abedf99\" (UID: \"33088a46-fcc7-48f1-90c4-43fc1abedf99\") " Jul 09 04:29:39.163646 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:39.163626 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33088a46-fcc7-48f1-90c4-43fc1abedf99-openshift-service-ca\") pod \"33088a46-fcc7-48f1-90c4-43fc1abedf99\" (UID: \"33088a46-fcc7-48f1-90c4-43fc1abedf99\") " Jul 09 04:29:39.163802 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:39.163648 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7txc\" (UniqueName: \"kubernetes.io/projected/33088a46-fcc7-48f1-90c4-43fc1abedf99-kube-api-access-t7txc\") pod \"33088a46-fcc7-48f1-90c4-43fc1abedf99\" (UID: \"33088a46-fcc7-48f1-90c4-43fc1abedf99\") " Jul 09 04:29:39.164034 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:39.164008 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33088a46-fcc7-48f1-90c4-43fc1abedf99-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33088a46-fcc7-48f1-90c4-43fc1abedf99" (UID: "33088a46-fcc7-48f1-90c4-43fc1abedf99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:29:39.165559 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:39.165537 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33088a46-fcc7-48f1-90c4-43fc1abedf99-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33088a46-fcc7-48f1-90c4-43fc1abedf99" (UID: "33088a46-fcc7-48f1-90c4-43fc1abedf99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:29:39.165649 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:39.165627 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33088a46-fcc7-48f1-90c4-43fc1abedf99-kube-api-access-t7txc" (OuterVolumeSpecName: "kube-api-access-t7txc") pod "33088a46-fcc7-48f1-90c4-43fc1abedf99" (UID: "33088a46-fcc7-48f1-90c4-43fc1abedf99"). InnerVolumeSpecName "kube-api-access-t7txc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:29:39.264199 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:39.264176 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33088a46-fcc7-48f1-90c4-43fc1abedf99-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:29:39.264199 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:39.264196 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33088a46-fcc7-48f1-90c4-43fc1abedf99-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:29:39.264316 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:39.264206 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7txc\" (UniqueName: \"kubernetes.io/projected/33088a46-fcc7-48f1-90c4-43fc1abedf99-kube-api-access-t7txc\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:29:40.004204 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:40.004139 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" Jul 09 04:29:40.004204 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:40.004174 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4" event={"ID":"33088a46-fcc7-48f1-90c4-43fc1abedf99","Type":"ContainerDied","Data":"b583eba26eb6e0fa484bb49259f0b8edd5c67578dea9bb283a41623b29f8fb57"} Jul 09 04:29:40.004664 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:40.004256 2571 scope.go:117] "RemoveContainer" containerID="86b988ef72e3a9764c44409ad126a5027edbc8ecd983fd901892774fc7a854a4" Jul 09 04:29:40.019990 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:40.019969 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4"] Jul 09 04:29:40.023316 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:40.023294 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2cb3ca-7dc95586f4-mf9b4"] Jul 09 04:29:41.625935 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:41.625907 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33088a46-fcc7-48f1-90c4-43fc1abedf99" path="/var/lib/kubelet/pods/33088a46-fcc7-48f1-90c4-43fc1abedf99/volumes" Jul 09 04:29:41.638218 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:41.638190 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:29:45.759839 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:45.759811 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg"] Jul 09 04:29:45.760264 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:45.760014 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" podUID="b269836e-e387-454b-9e53-b73aa51b6ea6" containerName="maas-api" containerID="cri-o://e92d1d54e193a72609dce209bd3d5a51a47a2f8ae8347001106c6ae414060c42" gracePeriod=30 Jul 09 04:29:46.026325 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.026299 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" Jul 09 04:29:46.036139 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.036103 2571 generic.go:356] "Generic (PLEG): container finished" podID="b269836e-e387-454b-9e53-b73aa51b6ea6" containerID="e92d1d54e193a72609dce209bd3d5a51a47a2f8ae8347001106c6ae414060c42" exitCode=0 Jul 09 04:29:46.036241 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.036157 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" Jul 09 04:29:46.036241 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.036188 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" event={"ID":"b269836e-e387-454b-9e53-b73aa51b6ea6","Type":"ContainerDied","Data":"e92d1d54e193a72609dce209bd3d5a51a47a2f8ae8347001106c6ae414060c42"} Jul 09 04:29:46.036323 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.036263 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg" event={"ID":"b269836e-e387-454b-9e53-b73aa51b6ea6","Type":"ContainerDied","Data":"f7c6df8e7d24c73fd59a8eda0d8e3ee6df76666e5b917b11616f58f116c38cc7"} Jul 09 04:29:46.036323 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.036288 2571 scope.go:117] "RemoveContainer" containerID="e92d1d54e193a72609dce209bd3d5a51a47a2f8ae8347001106c6ae414060c42" Jul 09 04:29:46.045940 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.045916 2571 scope.go:117] "RemoveContainer" containerID="e92d1d54e193a72609dce209bd3d5a51a47a2f8ae8347001106c6ae414060c42" Jul 09 04:29:46.046205 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:29:46.046181 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92d1d54e193a72609dce209bd3d5a51a47a2f8ae8347001106c6ae414060c42\": container with ID starting with e92d1d54e193a72609dce209bd3d5a51a47a2f8ae8347001106c6ae414060c42 not found: ID does not exist" containerID="e92d1d54e193a72609dce209bd3d5a51a47a2f8ae8347001106c6ae414060c42" Jul 09 04:29:46.046297 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.046208 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92d1d54e193a72609dce209bd3d5a51a47a2f8ae8347001106c6ae414060c42"} err="failed to get container status \"e92d1d54e193a72609dce209bd3d5a51a47a2f8ae8347001106c6ae414060c42\": rpc error: code = NotFound desc = could not find container \"e92d1d54e193a72609dce209bd3d5a51a47a2f8ae8347001106c6ae414060c42\": container with ID starting with e92d1d54e193a72609dce209bd3d5a51a47a2f8ae8347001106c6ae414060c42 not found: ID does not exist" Jul 09 04:29:46.122629 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.122612 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b269836e-e387-454b-9e53-b73aa51b6ea6-openshift-service-ca\") pod \"b269836e-e387-454b-9e53-b73aa51b6ea6\" (UID: \"b269836e-e387-454b-9e53-b73aa51b6ea6\") " Jul 09 04:29:46.122727 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.122647 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gggt\" (UniqueName: \"kubernetes.io/projected/b269836e-e387-454b-9e53-b73aa51b6ea6-kube-api-access-7gggt\") pod \"b269836e-e387-454b-9e53-b73aa51b6ea6\" (UID: \"b269836e-e387-454b-9e53-b73aa51b6ea6\") " Jul 09 04:29:46.122727 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.122663 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b269836e-e387-454b-9e53-b73aa51b6ea6-maas-api-tls\") pod \"b269836e-e387-454b-9e53-b73aa51b6ea6\" (UID: \"b269836e-e387-454b-9e53-b73aa51b6ea6\") " Jul 09 04:29:46.122949 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.122929 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b269836e-e387-454b-9e53-b73aa51b6ea6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b269836e-e387-454b-9e53-b73aa51b6ea6" (UID: "b269836e-e387-454b-9e53-b73aa51b6ea6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:29:46.124623 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.124600 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b269836e-e387-454b-9e53-b73aa51b6ea6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b269836e-e387-454b-9e53-b73aa51b6ea6" (UID: "b269836e-e387-454b-9e53-b73aa51b6ea6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:29:46.124623 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.124603 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b269836e-e387-454b-9e53-b73aa51b6ea6-kube-api-access-7gggt" (OuterVolumeSpecName: "kube-api-access-7gggt") pod "b269836e-e387-454b-9e53-b73aa51b6ea6" (UID: "b269836e-e387-454b-9e53-b73aa51b6ea6"). InnerVolumeSpecName "kube-api-access-7gggt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:29:46.223381 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.223356 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b269836e-e387-454b-9e53-b73aa51b6ea6-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:29:46.223381 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.223376 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gggt\" (UniqueName: \"kubernetes.io/projected/b269836e-e387-454b-9e53-b73aa51b6ea6-kube-api-access-7gggt\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:29:46.223381 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.223385 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b269836e-e387-454b-9e53-b73aa51b6ea6-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:29:46.356961 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.356936 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg"] Jul 09 04:29:46.361205 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:46.361186 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d61384-5fb9d58857-fqtvg"] Jul 09 04:29:47.625727 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:47.625690 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b269836e-e387-454b-9e53-b73aa51b6ea6" path="/var/lib/kubelet/pods/b269836e-e387-454b-9e53-b73aa51b6ea6/volumes" Jul 09 04:29:54.733132 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:54.733098 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:29:55.613309 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.613280 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7"] Jul 09 04:29:55.613659 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.613639 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33088a46-fcc7-48f1-90c4-43fc1abedf99" containerName="maas-api" Jul 09 04:29:55.613659 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.613654 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="33088a46-fcc7-48f1-90c4-43fc1abedf99" containerName="maas-api" Jul 09 04:29:55.613835 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.613672 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b269836e-e387-454b-9e53-b73aa51b6ea6" containerName="maas-api" Jul 09 04:29:55.613835 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.613677 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b269836e-e387-454b-9e53-b73aa51b6ea6" containerName="maas-api" Jul 09 04:29:55.613835 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.613744 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="33088a46-fcc7-48f1-90c4-43fc1abedf99" containerName="maas-api" Jul 09 04:29:55.613835 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.613752 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b269836e-e387-454b-9e53-b73aa51b6ea6" containerName="maas-api" Jul 09 04:29:55.618210 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.618192 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" Jul 09 04:29:55.620335 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.620313 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-6bd312-serving-cert\"" Jul 09 04:29:55.620658 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.620642 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 04:29:55.621410 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.621389 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 04:29:55.621410 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.621392 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-znghr\"" Jul 09 04:29:55.629347 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.629328 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7"] Jul 09 04:29:55.699719 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.699690 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44820edc-8793-4ccd-a870-0435882ae126-maas-api-tls\") pod \"maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7\" (UID: \"44820edc-8793-4ccd-a870-0435882ae126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" Jul 09 04:29:55.699889 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.699757 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44820edc-8793-4ccd-a870-0435882ae126-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7\" (UID: \"44820edc-8793-4ccd-a870-0435882ae126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" Jul 09 04:29:55.699889 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.699794 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9rb2\" (UniqueName: \"kubernetes.io/projected/44820edc-8793-4ccd-a870-0435882ae126-kube-api-access-b9rb2\") pod \"maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7\" (UID: \"44820edc-8793-4ccd-a870-0435882ae126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" Jul 09 04:29:55.800636 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.800599 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44820edc-8793-4ccd-a870-0435882ae126-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7\" (UID: \"44820edc-8793-4ccd-a870-0435882ae126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" Jul 09 04:29:55.801070 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.800663 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9rb2\" (UniqueName: \"kubernetes.io/projected/44820edc-8793-4ccd-a870-0435882ae126-kube-api-access-b9rb2\") pod \"maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7\" (UID: \"44820edc-8793-4ccd-a870-0435882ae126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" Jul 09 04:29:55.801070 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.800726 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44820edc-8793-4ccd-a870-0435882ae126-maas-api-tls\") pod \"maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7\" (UID: \"44820edc-8793-4ccd-a870-0435882ae126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" Jul 09 04:29:55.801070 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:29:55.800824 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-serving-cert: secret "maas-api-e2e-shared-a-6bd312-serving-cert" not found Jul 09 04:29:55.801070 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:29:55.800920 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44820edc-8793-4ccd-a870-0435882ae126-maas-api-tls podName:44820edc-8793-4ccd-a870-0435882ae126 nodeName:}" failed. No retries permitted until 2026-07-09 04:29:56.300899644 +0000 UTC m=+2031.159770722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44820edc-8793-4ccd-a870-0435882ae126-maas-api-tls") pod "maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" (UID: "44820edc-8793-4ccd-a870-0435882ae126") : secret "maas-api-e2e-shared-a-6bd312-serving-cert" not found Jul 09 04:29:55.801366 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.801340 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44820edc-8793-4ccd-a870-0435882ae126-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7\" (UID: \"44820edc-8793-4ccd-a870-0435882ae126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" Jul 09 04:29:55.811290 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:55.811267 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9rb2\" (UniqueName: \"kubernetes.io/projected/44820edc-8793-4ccd-a870-0435882ae126-kube-api-access-b9rb2\") pod \"maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7\" (UID: \"44820edc-8793-4ccd-a870-0435882ae126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" Jul 09 04:29:56.303562 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:56.303535 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44820edc-8793-4ccd-a870-0435882ae126-maas-api-tls\") pod \"maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7\" (UID: \"44820edc-8793-4ccd-a870-0435882ae126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" Jul 09 04:29:56.305994 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:56.305961 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44820edc-8793-4ccd-a870-0435882ae126-maas-api-tls\") pod \"maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7\" (UID: \"44820edc-8793-4ccd-a870-0435882ae126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" Jul 09 04:29:56.529743 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:56.529714 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" Jul 09 04:29:56.657004 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:56.656977 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7"] Jul 09 04:29:56.658280 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:29:56.658257 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44820edc_8793_4ccd_a870_0435882ae126.slice/crio-17adc32809e3ae3ba48c68a389812cc2e28ab5b55c501ffe0f96cac5d87641e9 WatchSource:0}: Error finding container 17adc32809e3ae3ba48c68a389812cc2e28ab5b55c501ffe0f96cac5d87641e9: Status 404 returned error can't find the container with id 17adc32809e3ae3ba48c68a389812cc2e28ab5b55c501ffe0f96cac5d87641e9 Jul 09 04:29:57.096864 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:57.096826 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" event={"ID":"44820edc-8793-4ccd-a870-0435882ae126","Type":"ContainerStarted","Data":"26455a1f362f917a37e5dbc0e366dd19ae656dcac429a1976f721f57892dd6f0"} Jul 09 04:29:57.097171 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:57.096896 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" event={"ID":"44820edc-8793-4ccd-a870-0435882ae126","Type":"ContainerStarted","Data":"17adc32809e3ae3ba48c68a389812cc2e28ab5b55c501ffe0f96cac5d87641e9"} Jul 09 04:29:57.097171 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:57.097012 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" Jul 09 04:29:57.114900 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:29:57.114828 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" podStartSLOduration=1.789400492 podStartE2EDuration="2.11481513s" totalImagesPullingTime="325.414638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:29:57.112251492 +0000 UTC m=+2031.971122591" watchObservedRunningTime="2026-07-09 04:29:57.11481513 +0000 UTC m=+2031.973686208" Jul 09 04:30:00.130780 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:00.130748 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc"] Jul 09 04:30:00.134639 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:00.134619 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc" Jul 09 04:30:00.142997 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:00.142976 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc"] Jul 09 04:30:00.238292 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:00.238254 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5b89\" (UniqueName: \"kubernetes.io/projected/e2a4058e-224b-4307-a5bf-750b6b5b6592-kube-api-access-f5b89\") pod \"maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc\" (UID: \"e2a4058e-224b-4307-a5bf-750b6b5b6592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc" Jul 09 04:30:00.339309 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:00.339271 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5b89\" (UniqueName: \"kubernetes.io/projected/e2a4058e-224b-4307-a5bf-750b6b5b6592-kube-api-access-f5b89\") pod \"maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc\" (UID: \"e2a4058e-224b-4307-a5bf-750b6b5b6592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc" Jul 09 04:30:00.348803 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:00.348777 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5b89\" (UniqueName: \"kubernetes.io/projected/e2a4058e-224b-4307-a5bf-750b6b5b6592-kube-api-access-f5b89\") pod \"maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc\" (UID: \"e2a4058e-224b-4307-a5bf-750b6b5b6592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc" Jul 09 04:30:00.445686 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:00.445623 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc" Jul 09 04:30:00.577219 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:00.577195 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc"] Jul 09 04:30:00.579025 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:30:00.578995 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a4058e_224b_4307_a5bf_750b6b5b6592.slice/crio-5693b0bee8dceba8332d4ee17ac4b773083e699edabc4fae7f92a8d98005681b WatchSource:0}: Error finding container 5693b0bee8dceba8332d4ee17ac4b773083e699edabc4fae7f92a8d98005681b: Status 404 returned error can't find the container with id 5693b0bee8dceba8332d4ee17ac4b773083e699edabc4fae7f92a8d98005681b Jul 09 04:30:01.120334 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:01.120301 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc" event={"ID":"e2a4058e-224b-4307-a5bf-750b6b5b6592","Type":"ContainerStarted","Data":"5693b0bee8dceba8332d4ee17ac4b773083e699edabc4fae7f92a8d98005681b"} Jul 09 04:30:03.108556 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:03.108516 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" Jul 09 04:30:04.138377 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:04.138339 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc" event={"ID":"e2a4058e-224b-4307-a5bf-750b6b5b6592","Type":"ContainerStarted","Data":"1e03d3b282411c43a61edacb46e78d59d6bfc80afbcbac557df2afc6f2703f0f"} Jul 09 04:30:04.153442 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:04.153397 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc" podStartSLOduration=1.004012915 podStartE2EDuration="4.153356922s" totalImagesPullingTime="3.149344007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:30:04.152274849 +0000 UTC m=+2039.011145948" watchObservedRunningTime="2026-07-09 04:30:04.153356922 +0000 UTC m=+2039.012228058" Jul 09 04:30:05.044737 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:05.044702 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:30:16.438576 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:16.438537 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:30:16.830321 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:16.830247 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr"] Jul 09 04:30:16.833845 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:16.833829 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:16.836064 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:16.836043 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6bd312\"/\"openshift-service-ca.crt\"" Jul 09 04:30:16.836740 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:16.836720 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6bd312\"/\"default-dockercfg-jdwn2\"" Jul 09 04:30:16.836849 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:16.836725 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6bd312\"/\"kube-root-ca.crt\"" Jul 09 04:30:16.837021 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:16.837009 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6bd312\"/\"auth-test-model-6bd312-kserve-self-signed-certs\"" Jul 09 04:30:16.843802 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:16.843784 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr"] Jul 09 04:30:16.973705 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:16.973677 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-tmp-dir\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:16.973896 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:16.973726 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdzpg\" (UniqueName: \"kubernetes.io/projected/509ef52a-942b-43d7-9853-ddefe06a49a7-kube-api-access-vdzpg\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:16.973896 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:16.973812 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509ef52a-942b-43d7-9853-ddefe06a49a7-tls-certs\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:16.973992 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:16.973927 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-model-cache\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:16.973992 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:16.973970 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-dshm\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:16.974066 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:16.973997 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-home\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:17.075128 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:17.075101 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509ef52a-942b-43d7-9853-ddefe06a49a7-tls-certs\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:17.075281 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:17.075165 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-model-cache\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:17.075348 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:17.075330 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-dshm\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:17.075396 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:17.075381 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-home\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:17.075468 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:17.075453 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-tmp-dir\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:17.075529 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:17.075483 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdzpg\" (UniqueName: \"kubernetes.io/projected/509ef52a-942b-43d7-9853-ddefe06a49a7-kube-api-access-vdzpg\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:17.075586 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:17.075546 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-model-cache\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:17.075728 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:17.075706 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-home\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:17.075851 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:17.075779 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-tmp-dir\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:17.077369 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:17.077350 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-dshm\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:17.077654 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:17.077631 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509ef52a-942b-43d7-9853-ddefe06a49a7-tls-certs\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:17.087672 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:17.087619 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdzpg\" (UniqueName: \"kubernetes.io/projected/509ef52a-942b-43d7-9853-ddefe06a49a7-kube-api-access-vdzpg\") pod \"auth-test-model-6bd312-kserve-fd44f8cff-gghkr\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:17.143186 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:17.143156 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:17.279437 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:17.279415 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr"] Jul 09 04:30:17.280226 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:30:17.280202 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod509ef52a_942b_43d7_9853_ddefe06a49a7.slice/crio-b6aa97c5c53779cf20b072824f3d61c421041d903b5e96441c0c0f859ff0f16d WatchSource:0}: Error finding container b6aa97c5c53779cf20b072824f3d61c421041d903b5e96441c0c0f859ff0f16d: Status 404 returned error can't find the container with id b6aa97c5c53779cf20b072824f3d61c421041d903b5e96441c0c0f859ff0f16d Jul 09 04:30:18.217333 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:18.217302 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" event={"ID":"509ef52a-942b-43d7-9853-ddefe06a49a7","Type":"ContainerStarted","Data":"0063ade7fb7efdce1c6f8f07f570510c1c13e0bc1da0483a7778ef8d81ee0926"} Jul 09 04:30:18.217333 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:18.217338 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" event={"ID":"509ef52a-942b-43d7-9853-ddefe06a49a7","Type":"ContainerStarted","Data":"b6aa97c5c53779cf20b072824f3d61c421041d903b5e96441c0c0f859ff0f16d"} Jul 09 04:30:18.233787 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:18.233730 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" podStartSLOduration=2.2337046 podStartE2EDuration="2.2337046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:30:18.233520985 +0000 UTC m=+2053.092392076" watchObservedRunningTime="2026-07-09 04:30:18.2337046 +0000 UTC m=+2053.092575710" Jul 09 04:30:26.939603 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:26.939566 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:30:27.143799 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:27.143718 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:27.143799 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:27.143771 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:27.155684 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:27.155657 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:27.272956 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:27.272884 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:30:39.340306 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.340158 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h"] Jul 09 04:30:39.345984 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.345960 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.348925 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.348904 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-589b59\"/\"default-dockercfg-wh6tv\"" Jul 09 04:30:39.349845 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.349824 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-589b59\"/\"kube-root-ca.crt\"" Jul 09 04:30:39.350127 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.349838 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-589b59\"/\"openshift-service-ca.crt\"" Jul 09 04:30:39.350319 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.350126 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-589b59\"/\"auth-test-model-589b59-kserve-self-signed-certs\"" Jul 09 04:30:39.363110 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.363084 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-model-cache\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.363183 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.363159 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183f6681-8015-4197-8b9d-cf05d82b0662-tls-certs\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.363219 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.363186 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92rq5\" (UniqueName: \"kubernetes.io/projected/183f6681-8015-4197-8b9d-cf05d82b0662-kube-api-access-92rq5\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.363316 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.363298 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-home\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.363370 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.363327 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-tmp-dir\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.363370 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.363344 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-dshm\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.365867 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.365674 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h"] Jul 09 04:30:39.463993 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.463951 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-home\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.463993 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.463997 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-tmp-dir\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.464252 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.464013 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-dshm\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.464252 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.464133 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-model-cache\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.464252 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.464222 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183f6681-8015-4197-8b9d-cf05d82b0662-tls-certs\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.464401 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.464253 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92rq5\" (UniqueName: \"kubernetes.io/projected/183f6681-8015-4197-8b9d-cf05d82b0662-kube-api-access-92rq5\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.464456 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.464405 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-home\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.464456 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.464428 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-tmp-dir\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.464549 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.464484 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-model-cache\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.466105 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.466079 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-dshm\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.466472 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.466454 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183f6681-8015-4197-8b9d-cf05d82b0662-tls-certs\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.473092 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.473072 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92rq5\" (UniqueName: \"kubernetes.io/projected/183f6681-8015-4197-8b9d-cf05d82b0662-kube-api-access-92rq5\") pod \"auth-test-model-589b59-kserve-69b444647-ctw9h\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.667927 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.667899 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:39.800803 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:39.800779 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h"] Jul 09 04:30:39.803328 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:30:39.803303 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod183f6681_8015_4197_8b9d_cf05d82b0662.slice/crio-8fe189ba02d5af9d732f56b5883ca4301ff29a21511aeefa39f54dfbbf64babb WatchSource:0}: Error finding container 8fe189ba02d5af9d732f56b5883ca4301ff29a21511aeefa39f54dfbbf64babb: Status 404 returned error can't find the container with id 8fe189ba02d5af9d732f56b5883ca4301ff29a21511aeefa39f54dfbbf64babb Jul 09 04:30:40.337115 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:40.337079 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" event={"ID":"183f6681-8015-4197-8b9d-cf05d82b0662","Type":"ContainerStarted","Data":"dee05f4203efd51dc5c11b3a9f39cf36f12ab6d90478cbe9a08771400ecdf793"} Jul 09 04:30:40.337299 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:40.337125 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" event={"ID":"183f6681-8015-4197-8b9d-cf05d82b0662","Type":"ContainerStarted","Data":"8fe189ba02d5af9d732f56b5883ca4301ff29a21511aeefa39f54dfbbf64babb"} Jul 09 04:30:40.352771 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:40.352730 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" podStartSLOduration=1.352715959 podStartE2EDuration="1.352715959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:30:40.351503084 +0000 UTC m=+2075.210374185" watchObservedRunningTime="2026-07-09 04:30:40.352715959 +0000 UTC m=+2075.211587057" Jul 09 04:30:43.635543 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:43.635513 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:30:49.668456 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:49.668424 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:49.668963 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:49.668470 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:49.680343 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:49.680317 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:30:50.398410 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:30:50.398382 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:31:00.014306 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:00.014270 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc"] Jul 09 04:31:00.014790 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:00.014529 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc" podUID="e2a4058e-224b-4307-a5bf-750b6b5b6592" containerName="cleanup" containerID="cri-o://1e03d3b282411c43a61edacb46e78d59d6bfc80afbcbac557df2afc6f2703f0f" gracePeriod=30 Jul 09 04:31:05.737928 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:05.737903 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:31:05.743470 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:05.743450 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:31:07.539531 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:07.539495 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:31:26.533324 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:26.533291 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:31:30.192829 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:30.192806 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc" Jul 09 04:31:30.197733 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:30.197712 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5b89\" (UniqueName: \"kubernetes.io/projected/e2a4058e-224b-4307-a5bf-750b6b5b6592-kube-api-access-f5b89\") pod \"e2a4058e-224b-4307-a5bf-750b6b5b6592\" (UID: \"e2a4058e-224b-4307-a5bf-750b6b5b6592\") " Jul 09 04:31:30.199539 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:30.199519 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a4058e-224b-4307-a5bf-750b6b5b6592-kube-api-access-f5b89" (OuterVolumeSpecName: "kube-api-access-f5b89") pod "e2a4058e-224b-4307-a5bf-750b6b5b6592" (UID: "e2a4058e-224b-4307-a5bf-750b6b5b6592"). InnerVolumeSpecName "kube-api-access-f5b89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:31:30.298389 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:30.298313 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5b89\" (UniqueName: \"kubernetes.io/projected/e2a4058e-224b-4307-a5bf-750b6b5b6592-kube-api-access-f5b89\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:31:30.597109 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:30.597073 2571 generic.go:356] "Generic (PLEG): container finished" podID="e2a4058e-224b-4307-a5bf-750b6b5b6592" containerID="1e03d3b282411c43a61edacb46e78d59d6bfc80afbcbac557df2afc6f2703f0f" exitCode=137 Jul 09 04:31:30.597242 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:30.597119 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc" Jul 09 04:31:30.597242 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:30.597159 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc" event={"ID":"e2a4058e-224b-4307-a5bf-750b6b5b6592","Type":"ContainerDied","Data":"1e03d3b282411c43a61edacb46e78d59d6bfc80afbcbac557df2afc6f2703f0f"} Jul 09 04:31:30.597242 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:30.597212 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc" event={"ID":"e2a4058e-224b-4307-a5bf-750b6b5b6592","Type":"ContainerDied","Data":"5693b0bee8dceba8332d4ee17ac4b773083e699edabc4fae7f92a8d98005681b"} Jul 09 04:31:30.597242 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:30.597235 2571 scope.go:117] "RemoveContainer" containerID="1e03d3b282411c43a61edacb46e78d59d6bfc80afbcbac557df2afc6f2703f0f" Jul 09 04:31:30.606098 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:30.606081 2571 scope.go:117] "RemoveContainer" containerID="1e03d3b282411c43a61edacb46e78d59d6bfc80afbcbac557df2afc6f2703f0f" Jul 09 04:31:30.606324 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:31:30.606308 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e03d3b282411c43a61edacb46e78d59d6bfc80afbcbac557df2afc6f2703f0f\": container with ID starting with 1e03d3b282411c43a61edacb46e78d59d6bfc80afbcbac557df2afc6f2703f0f not found: ID does not exist" containerID="1e03d3b282411c43a61edacb46e78d59d6bfc80afbcbac557df2afc6f2703f0f" Jul 09 04:31:30.606362 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:30.606335 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e03d3b282411c43a61edacb46e78d59d6bfc80afbcbac557df2afc6f2703f0f"} err="failed to get container status \"1e03d3b282411c43a61edacb46e78d59d6bfc80afbcbac557df2afc6f2703f0f\": rpc error: code = NotFound desc = could not find container \"1e03d3b282411c43a61edacb46e78d59d6bfc80afbcbac557df2afc6f2703f0f\": container with ID starting with 1e03d3b282411c43a61edacb46e78d59d6bfc80afbcbac557df2afc6f2703f0f not found: ID does not exist" Jul 09 04:31:30.621124 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:30.621095 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc"] Jul 09 04:31:30.625758 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:30.625735 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bd312-29726190-cdbwc"] Jul 09 04:31:31.625546 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:31.625519 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2a4058e-224b-4307-a5bf-750b6b5b6592" path="/var/lib/kubelet/pods/e2a4058e-224b-4307-a5bf-750b6b5b6592/volumes" Jul 09 04:31:45.442316 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:31:45.442285 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:32:00.633179 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:32:00.633102 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:32:17.839731 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:32:17.839695 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:32:34.041867 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:32:34.041818 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:32:53.736741 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:32:53.736702 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:32:57.587817 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:32:57.587788 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr"] Jul 09 04:32:58.546808 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:32:58.546776 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h"] Jul 09 04:33:07.644776 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.644746 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p"] Jul 09 04:33:07.645298 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.645107 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a4058e-224b-4307-a5bf-750b6b5b6592" containerName="cleanup" Jul 09 04:33:07.645298 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.645118 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a4058e-224b-4307-a5bf-750b6b5b6592" containerName="cleanup" Jul 09 04:33:07.645298 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.645185 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a4058e-224b-4307-a5bf-750b6b5b6592" containerName="cleanup" Jul 09 04:33:07.648170 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.648154 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.650383 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.650359 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6bd312\"/\"sub-test-model-6bd312-kserve-self-signed-certs\"" Jul 09 04:33:07.656673 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.656656 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p"] Jul 09 04:33:07.791146 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.791114 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-home\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.791146 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.791149 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/579df4c9-299a-400b-b783-a7999380d127-tls-certs\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.791297 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.791223 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-dshm\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.791297 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.791257 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-model-cache\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.791297 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.791283 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lk74\" (UniqueName: \"kubernetes.io/projected/579df4c9-299a-400b-b783-a7999380d127-kube-api-access-4lk74\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.791397 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.791348 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-tmp-dir\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.892252 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.892224 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-tmp-dir\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.892348 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.892263 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-home\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.892348 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.892282 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/579df4c9-299a-400b-b783-a7999380d127-tls-certs\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.892348 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.892333 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-dshm\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.892483 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.892353 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-model-cache\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.892483 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.892369 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lk74\" (UniqueName: \"kubernetes.io/projected/579df4c9-299a-400b-b783-a7999380d127-kube-api-access-4lk74\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.892699 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.892677 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-tmp-dir\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.892771 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.892712 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-home\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.892771 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.892731 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-model-cache\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.894537 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.894515 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-dshm\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.894809 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.894788 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/579df4c9-299a-400b-b783-a7999380d127-tls-certs\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.899597 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.899577 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lk74\" (UniqueName: \"kubernetes.io/projected/579df4c9-299a-400b-b783-a7999380d127-kube-api-access-4lk74\") pod \"sub-test-model-6bd312-kserve-795d84d5d7-hl27p\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:07.957259 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:07.957236 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:08.086148 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:08.086122 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p"] Jul 09 04:33:08.087185 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:33:08.087156 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod579df4c9_299a_400b_b783_a7999380d127.slice/crio-29d7a833e0fbff7411d59a19712b77d81de6d7ee8e9423ef7f82a37b311971cb WatchSource:0}: Error finding container 29d7a833e0fbff7411d59a19712b77d81de6d7ee8e9423ef7f82a37b311971cb: Status 404 returned error can't find the container with id 29d7a833e0fbff7411d59a19712b77d81de6d7ee8e9423ef7f82a37b311971cb Jul 09 04:33:08.097103 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:08.097079 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" event={"ID":"579df4c9-299a-400b-b783-a7999380d127","Type":"ContainerStarted","Data":"29d7a833e0fbff7411d59a19712b77d81de6d7ee8e9423ef7f82a37b311971cb"} Jul 09 04:33:09.104568 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:09.104536 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" event={"ID":"579df4c9-299a-400b-b783-a7999380d127","Type":"ContainerStarted","Data":"edfc13286c263c6536befcb97cc95e044c7813c0d4adb9fe86eca4bf3c621a4a"} Jul 09 04:33:09.123119 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:09.123075 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" podStartSLOduration=2.12306134 podStartE2EDuration="2.12306134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:33:09.120279657 +0000 UTC m=+2223.979150767" watchObservedRunningTime="2026-07-09 04:33:09.12306134 +0000 UTC m=+2223.981932438" Jul 09 04:33:10.540808 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:10.540775 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:33:12.607209 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:12.607170 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" podUID="509ef52a-942b-43d7-9853-ddefe06a49a7" containerName="main" containerID="cri-o://0063ade7fb7efdce1c6f8f07f570510c1c13e0bc1da0483a7778ef8d81ee0926" gracePeriod=45 Jul 09 04:33:12.851605 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:12.851573 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:33:12.934209 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:12.934150 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-dshm\") pod \"509ef52a-942b-43d7-9853-ddefe06a49a7\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " Jul 09 04:33:12.934209 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:12.934196 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdzpg\" (UniqueName: \"kubernetes.io/projected/509ef52a-942b-43d7-9853-ddefe06a49a7-kube-api-access-vdzpg\") pod \"509ef52a-942b-43d7-9853-ddefe06a49a7\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " Jul 09 04:33:12.934357 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:12.934235 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-tmp-dir\") pod \"509ef52a-942b-43d7-9853-ddefe06a49a7\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " Jul 09 04:33:12.934357 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:12.934257 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509ef52a-942b-43d7-9853-ddefe06a49a7-tls-certs\") pod \"509ef52a-942b-43d7-9853-ddefe06a49a7\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " Jul 09 04:33:12.934357 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:12.934285 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-model-cache\") pod \"509ef52a-942b-43d7-9853-ddefe06a49a7\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " Jul 09 04:33:12.934357 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:12.934304 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-home\") pod \"509ef52a-942b-43d7-9853-ddefe06a49a7\" (UID: \"509ef52a-942b-43d7-9853-ddefe06a49a7\") " Jul 09 04:33:12.934568 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:12.934480 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "509ef52a-942b-43d7-9853-ddefe06a49a7" (UID: "509ef52a-942b-43d7-9853-ddefe06a49a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:33:12.934568 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:12.934531 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "509ef52a-942b-43d7-9853-ddefe06a49a7" (UID: "509ef52a-942b-43d7-9853-ddefe06a49a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:33:12.934667 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:12.934615 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-home" (OuterVolumeSpecName: "home") pod "509ef52a-942b-43d7-9853-ddefe06a49a7" (UID: "509ef52a-942b-43d7-9853-ddefe06a49a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:33:12.936234 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:12.936205 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-dshm" (OuterVolumeSpecName: "dshm") pod "509ef52a-942b-43d7-9853-ddefe06a49a7" (UID: "509ef52a-942b-43d7-9853-ddefe06a49a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:33:12.936326 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:12.936275 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509ef52a-942b-43d7-9853-ddefe06a49a7-kube-api-access-vdzpg" (OuterVolumeSpecName: "kube-api-access-vdzpg") pod "509ef52a-942b-43d7-9853-ddefe06a49a7" (UID: "509ef52a-942b-43d7-9853-ddefe06a49a7"). InnerVolumeSpecName "kube-api-access-vdzpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:33:12.936326 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:12.936275 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/509ef52a-942b-43d7-9853-ddefe06a49a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "509ef52a-942b-43d7-9853-ddefe06a49a7" (UID: "509ef52a-942b-43d7-9853-ddefe06a49a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:33:13.035278 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.035253 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdzpg\" (UniqueName: \"kubernetes.io/projected/509ef52a-942b-43d7-9853-ddefe06a49a7-kube-api-access-vdzpg\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:33:13.035394 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.035290 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-tmp-dir\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:33:13.035394 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.035300 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509ef52a-942b-43d7-9853-ddefe06a49a7-tls-certs\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:33:13.035394 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.035309 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-model-cache\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:33:13.035394 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.035316 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-home\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:33:13.035394 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.035323 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509ef52a-942b-43d7-9853-ddefe06a49a7-dshm\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:33:13.128531 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.128505 2571 generic.go:356] "Generic (PLEG): container finished" podID="509ef52a-942b-43d7-9853-ddefe06a49a7" containerID="0063ade7fb7efdce1c6f8f07f570510c1c13e0bc1da0483a7778ef8d81ee0926" exitCode=0 Jul 09 04:33:13.128634 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.128552 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" Jul 09 04:33:13.128634 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.128588 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" event={"ID":"509ef52a-942b-43d7-9853-ddefe06a49a7","Type":"ContainerDied","Data":"0063ade7fb7efdce1c6f8f07f570510c1c13e0bc1da0483a7778ef8d81ee0926"} Jul 09 04:33:13.128634 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.128628 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr" event={"ID":"509ef52a-942b-43d7-9853-ddefe06a49a7","Type":"ContainerDied","Data":"b6aa97c5c53779cf20b072824f3d61c421041d903b5e96441c0c0f859ff0f16d"} Jul 09 04:33:13.128743 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.128644 2571 scope.go:117] "RemoveContainer" containerID="0063ade7fb7efdce1c6f8f07f570510c1c13e0bc1da0483a7778ef8d81ee0926" Jul 09 04:33:13.137355 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.137340 2571 scope.go:117] "RemoveContainer" containerID="0063ade7fb7efdce1c6f8f07f570510c1c13e0bc1da0483a7778ef8d81ee0926" Jul 09 04:33:13.137588 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:33:13.137571 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0063ade7fb7efdce1c6f8f07f570510c1c13e0bc1da0483a7778ef8d81ee0926\": container with ID starting with 0063ade7fb7efdce1c6f8f07f570510c1c13e0bc1da0483a7778ef8d81ee0926 not found: ID does not exist" containerID="0063ade7fb7efdce1c6f8f07f570510c1c13e0bc1da0483a7778ef8d81ee0926" Jul 09 04:33:13.137637 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.137596 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0063ade7fb7efdce1c6f8f07f570510c1c13e0bc1da0483a7778ef8d81ee0926"} err="failed to get container status \"0063ade7fb7efdce1c6f8f07f570510c1c13e0bc1da0483a7778ef8d81ee0926\": rpc error: code = NotFound desc = could not find container \"0063ade7fb7efdce1c6f8f07f570510c1c13e0bc1da0483a7778ef8d81ee0926\": container with ID starting with 0063ade7fb7efdce1c6f8f07f570510c1c13e0bc1da0483a7778ef8d81ee0926 not found: ID does not exist" Jul 09 04:33:13.148768 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.148740 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr"] Jul 09 04:33:13.154244 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.154224 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/auth-test-model-6bd312-kserve-fd44f8cff-gghkr"] Jul 09 04:33:13.566611 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.566576 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" podUID="183f6681-8015-4197-8b9d-cf05d82b0662" containerName="main" containerID="cri-o://dee05f4203efd51dc5c11b3a9f39cf36f12ab6d90478cbe9a08771400ecdf793" gracePeriod=45 Jul 09 04:33:13.626892 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.626844 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="509ef52a-942b-43d7-9853-ddefe06a49a7" path="/var/lib/kubelet/pods/509ef52a-942b-43d7-9853-ddefe06a49a7/volumes" Jul 09 04:33:13.808376 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.808355 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:33:13.943589 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.943568 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183f6681-8015-4197-8b9d-cf05d82b0662-tls-certs\") pod \"183f6681-8015-4197-8b9d-cf05d82b0662\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " Jul 09 04:33:13.943743 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.943611 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-model-cache\") pod \"183f6681-8015-4197-8b9d-cf05d82b0662\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " Jul 09 04:33:13.943743 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.943637 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-tmp-dir\") pod \"183f6681-8015-4197-8b9d-cf05d82b0662\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " Jul 09 04:33:13.943743 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.943684 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92rq5\" (UniqueName: \"kubernetes.io/projected/183f6681-8015-4197-8b9d-cf05d82b0662-kube-api-access-92rq5\") pod \"183f6681-8015-4197-8b9d-cf05d82b0662\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " Jul 09 04:33:13.943938 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.943750 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-dshm\") pod \"183f6681-8015-4197-8b9d-cf05d82b0662\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " Jul 09 04:33:13.943938 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.943828 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-model-cache" (OuterVolumeSpecName: "model-cache") pod "183f6681-8015-4197-8b9d-cf05d82b0662" (UID: "183f6681-8015-4197-8b9d-cf05d82b0662"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:33:13.943938 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.943902 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-home\") pod \"183f6681-8015-4197-8b9d-cf05d82b0662\" (UID: \"183f6681-8015-4197-8b9d-cf05d82b0662\") " Jul 09 04:33:13.944107 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.943932 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "183f6681-8015-4197-8b9d-cf05d82b0662" (UID: "183f6681-8015-4197-8b9d-cf05d82b0662"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:33:13.944167 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.944131 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-home" (OuterVolumeSpecName: "home") pod "183f6681-8015-4197-8b9d-cf05d82b0662" (UID: "183f6681-8015-4197-8b9d-cf05d82b0662"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:33:13.944218 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.944196 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-model-cache\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:33:13.944218 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.944211 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-tmp-dir\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:33:13.944324 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.944224 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-home\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:33:13.945720 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.945702 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/183f6681-8015-4197-8b9d-cf05d82b0662-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "183f6681-8015-4197-8b9d-cf05d82b0662" (UID: "183f6681-8015-4197-8b9d-cf05d82b0662"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:33:13.945811 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.945777 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183f6681-8015-4197-8b9d-cf05d82b0662-kube-api-access-92rq5" (OuterVolumeSpecName: "kube-api-access-92rq5") pod "183f6681-8015-4197-8b9d-cf05d82b0662" (UID: "183f6681-8015-4197-8b9d-cf05d82b0662"). InnerVolumeSpecName "kube-api-access-92rq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:33:13.945911 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:13.945893 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-dshm" (OuterVolumeSpecName: "dshm") pod "183f6681-8015-4197-8b9d-cf05d82b0662" (UID: "183f6681-8015-4197-8b9d-cf05d82b0662"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:33:14.045533 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:14.045516 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183f6681-8015-4197-8b9d-cf05d82b0662-tls-certs\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:33:14.045533 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:14.045534 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92rq5\" (UniqueName: \"kubernetes.io/projected/183f6681-8015-4197-8b9d-cf05d82b0662-kube-api-access-92rq5\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:33:14.045666 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:14.045544 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183f6681-8015-4197-8b9d-cf05d82b0662-dshm\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:33:14.135900 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:14.135875 2571 generic.go:356] "Generic (PLEG): container finished" podID="183f6681-8015-4197-8b9d-cf05d82b0662" containerID="dee05f4203efd51dc5c11b3a9f39cf36f12ab6d90478cbe9a08771400ecdf793" exitCode=0 Jul 09 04:33:14.135994 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:14.135912 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" Jul 09 04:33:14.135994 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:14.135953 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" event={"ID":"183f6681-8015-4197-8b9d-cf05d82b0662","Type":"ContainerDied","Data":"dee05f4203efd51dc5c11b3a9f39cf36f12ab6d90478cbe9a08771400ecdf793"} Jul 09 04:33:14.136078 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:14.135994 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h" event={"ID":"183f6681-8015-4197-8b9d-cf05d82b0662","Type":"ContainerDied","Data":"8fe189ba02d5af9d732f56b5883ca4301ff29a21511aeefa39f54dfbbf64babb"} Jul 09 04:33:14.136078 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:14.136016 2571 scope.go:117] "RemoveContainer" containerID="dee05f4203efd51dc5c11b3a9f39cf36f12ab6d90478cbe9a08771400ecdf793" Jul 09 04:33:14.144568 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:14.144555 2571 scope.go:117] "RemoveContainer" containerID="dee05f4203efd51dc5c11b3a9f39cf36f12ab6d90478cbe9a08771400ecdf793" Jul 09 04:33:14.144798 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:33:14.144781 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee05f4203efd51dc5c11b3a9f39cf36f12ab6d90478cbe9a08771400ecdf793\": container with ID starting with dee05f4203efd51dc5c11b3a9f39cf36f12ab6d90478cbe9a08771400ecdf793 not found: ID does not exist" containerID="dee05f4203efd51dc5c11b3a9f39cf36f12ab6d90478cbe9a08771400ecdf793" Jul 09 04:33:14.144871 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:14.144808 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee05f4203efd51dc5c11b3a9f39cf36f12ab6d90478cbe9a08771400ecdf793"} err="failed to get container status \"dee05f4203efd51dc5c11b3a9f39cf36f12ab6d90478cbe9a08771400ecdf793\": rpc error: code = NotFound desc = could not find container \"dee05f4203efd51dc5c11b3a9f39cf36f12ab6d90478cbe9a08771400ecdf793\": container with ID starting with dee05f4203efd51dc5c11b3a9f39cf36f12ab6d90478cbe9a08771400ecdf793 not found: ID does not exist" Jul 09 04:33:14.156293 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:14.156271 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h"] Jul 09 04:33:14.158268 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:14.158251 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-589b59/auth-test-model-589b59-kserve-69b444647-ctw9h"] Jul 09 04:33:15.626077 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:15.626046 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="183f6681-8015-4197-8b9d-cf05d82b0662" path="/var/lib/kubelet/pods/183f6681-8015-4197-8b9d-cf05d82b0662/volumes" Jul 09 04:33:17.958341 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:17.958311 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:17.958701 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:17.958353 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:17.970209 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:17.970188 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:33:18.168631 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:33:18.168605 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:34:04.546583 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.546547 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8"] Jul 09 04:34:04.547894 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.547870 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="509ef52a-942b-43d7-9853-ddefe06a49a7" containerName="main" Jul 09 04:34:04.548051 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.548040 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="509ef52a-942b-43d7-9853-ddefe06a49a7" containerName="main" Jul 09 04:34:04.548136 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.548127 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="183f6681-8015-4197-8b9d-cf05d82b0662" containerName="main" Jul 09 04:34:04.548202 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.548194 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="183f6681-8015-4197-8b9d-cf05d82b0662" containerName="main" Jul 09 04:34:04.548476 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.548463 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="183f6681-8015-4197-8b9d-cf05d82b0662" containerName="main" Jul 09 04:34:04.548588 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.548579 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="509ef52a-942b-43d7-9853-ddefe06a49a7" containerName="main" Jul 09 04:34:04.552486 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.552461 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.554829 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.554808 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8"] Jul 09 04:34:04.555435 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.555413 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-589b59\"/\"sub-test-model-589b59-kserve-self-signed-certs\"" Jul 09 04:34:04.555588 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.555413 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-589b59\"/\"openshift-service-ca.crt\"" Jul 09 04:34:04.555748 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.555413 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-589b59\"/\"default-dockercfg-wh6tv\"" Jul 09 04:34:04.555821 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.555492 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-589b59\"/\"kube-root-ca.crt\"" Jul 09 04:34:04.636722 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.636696 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eaefb4e-473f-41f5-a4cd-f6d32d196957-tls-certs\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.636825 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.636738 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-model-cache\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.636825 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.636769 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-dshm\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.636825 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.636791 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf2tj\" (UniqueName: \"kubernetes.io/projected/9eaefb4e-473f-41f5-a4cd-f6d32d196957-kube-api-access-nf2tj\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.636825 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.636808 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-home\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.637012 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.636827 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-tmp-dir\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.737418 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.737392 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eaefb4e-473f-41f5-a4cd-f6d32d196957-tls-certs\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.737517 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.737429 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-model-cache\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.737517 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.737465 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-dshm\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.737517 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.737489 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf2tj\" (UniqueName: \"kubernetes.io/projected/9eaefb4e-473f-41f5-a4cd-f6d32d196957-kube-api-access-nf2tj\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.737517 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.737512 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-home\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.737709 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.737544 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-tmp-dir\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.737849 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.737825 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-model-cache\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.737978 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.737958 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-home\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.738030 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.737978 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-tmp-dir\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.739605 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.739576 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-dshm\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.739837 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.739821 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eaefb4e-473f-41f5-a4cd-f6d32d196957-tls-certs\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.745293 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.745268 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf2tj\" (UniqueName: \"kubernetes.io/projected/9eaefb4e-473f-41f5-a4cd-f6d32d196957-kube-api-access-nf2tj\") pod \"sub-test-model-589b59-kserve-6ff5f69d4c-9swx8\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:04.863566 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:04.863540 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:05.016813 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:05.016789 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8"] Jul 09 04:34:05.018738 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:34:05.018713 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9eaefb4e_473f_41f5_a4cd_f6d32d196957.slice/crio-56c283a2af6f2d5542c7d482dee6a6c0440b0e4a5faa1044c7143e78f3ac4974 WatchSource:0}: Error finding container 56c283a2af6f2d5542c7d482dee6a6c0440b0e4a5faa1044c7143e78f3ac4974: Status 404 returned error can't find the container with id 56c283a2af6f2d5542c7d482dee6a6c0440b0e4a5faa1044c7143e78f3ac4974 Jul 09 04:34:05.400713 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:05.400671 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" event={"ID":"9eaefb4e-473f-41f5-a4cd-f6d32d196957","Type":"ContainerStarted","Data":"f8f4cfdf21fc467404d8a51b04e51e1b7e573b3a53eace8f2c5699f2bf7baee6"} Jul 09 04:34:05.400891 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:05.400719 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" event={"ID":"9eaefb4e-473f-41f5-a4cd-f6d32d196957","Type":"ContainerStarted","Data":"56c283a2af6f2d5542c7d482dee6a6c0440b0e4a5faa1044c7143e78f3ac4974"} Jul 09 04:34:05.417902 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:05.417836 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" podStartSLOduration=1.417821968 podStartE2EDuration="1.417821968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:34:05.41578502 +0000 UTC m=+2280.274656119" watchObservedRunningTime="2026-07-09 04:34:05.417821968 +0000 UTC m=+2280.276693065" Jul 09 04:34:09.647026 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:09.646994 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:34:14.864480 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:14.864447 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:14.864909 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:14.864497 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:14.876609 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:14.876586 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:15.462800 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:15.462764 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:25.832484 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:25.832453 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:34:41.744626 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:41.744595 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p"] Jul 09 04:34:42.638461 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:42.638432 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8"] Jul 09 04:34:46.535276 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:46.535247 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:34:56.763429 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:56.763371 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" podUID="579df4c9-299a-400b-b783-a7999380d127" containerName="main" containerID="cri-o://edfc13286c263c6536befcb97cc95e044c7813c0d4adb9fe86eca4bf3c621a4a" gracePeriod=45 Jul 09 04:34:57.006282 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.006257 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:34:57.034536 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.034449 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lk74\" (UniqueName: \"kubernetes.io/projected/579df4c9-299a-400b-b783-a7999380d127-kube-api-access-4lk74\") pod \"579df4c9-299a-400b-b783-a7999380d127\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " Jul 09 04:34:57.034536 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.034500 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-dshm\") pod \"579df4c9-299a-400b-b783-a7999380d127\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " Jul 09 04:34:57.034701 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.034537 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/579df4c9-299a-400b-b783-a7999380d127-tls-certs\") pod \"579df4c9-299a-400b-b783-a7999380d127\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " Jul 09 04:34:57.034701 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.034597 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-model-cache\") pod \"579df4c9-299a-400b-b783-a7999380d127\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " Jul 09 04:34:57.034701 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.034621 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-home\") pod \"579df4c9-299a-400b-b783-a7999380d127\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " Jul 09 04:34:57.034701 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.034647 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-tmp-dir\") pod \"579df4c9-299a-400b-b783-a7999380d127\" (UID: \"579df4c9-299a-400b-b783-a7999380d127\") " Jul 09 04:34:57.034954 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.034866 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-model-cache" (OuterVolumeSpecName: "model-cache") pod "579df4c9-299a-400b-b783-a7999380d127" (UID: "579df4c9-299a-400b-b783-a7999380d127"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:34:57.034954 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.034890 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-home" (OuterVolumeSpecName: "home") pod "579df4c9-299a-400b-b783-a7999380d127" (UID: "579df4c9-299a-400b-b783-a7999380d127"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:34:57.035072 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.034980 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-model-cache\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:34:57.035072 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.034991 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-home\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:34:57.035072 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.035057 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "579df4c9-299a-400b-b783-a7999380d127" (UID: "579df4c9-299a-400b-b783-a7999380d127"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:34:57.036578 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.036553 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-dshm" (OuterVolumeSpecName: "dshm") pod "579df4c9-299a-400b-b783-a7999380d127" (UID: "579df4c9-299a-400b-b783-a7999380d127"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:34:57.036992 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.036963 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/579df4c9-299a-400b-b783-a7999380d127-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "579df4c9-299a-400b-b783-a7999380d127" (UID: "579df4c9-299a-400b-b783-a7999380d127"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:34:57.037071 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.037004 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/579df4c9-299a-400b-b783-a7999380d127-kube-api-access-4lk74" (OuterVolumeSpecName: "kube-api-access-4lk74") pod "579df4c9-299a-400b-b783-a7999380d127" (UID: "579df4c9-299a-400b-b783-a7999380d127"). InnerVolumeSpecName "kube-api-access-4lk74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:34:57.136236 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.136212 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/579df4c9-299a-400b-b783-a7999380d127-tls-certs\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:34:57.136236 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.136236 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-tmp-dir\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:34:57.136364 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.136248 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lk74\" (UniqueName: \"kubernetes.io/projected/579df4c9-299a-400b-b783-a7999380d127-kube-api-access-4lk74\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:34:57.136364 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.136259 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/579df4c9-299a-400b-b783-a7999380d127-dshm\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:34:57.657791 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.657738 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" podUID="9eaefb4e-473f-41f5-a4cd-f6d32d196957" containerName="main" containerID="cri-o://f8f4cfdf21fc467404d8a51b04e51e1b7e573b3a53eace8f2c5699f2bf7baee6" gracePeriod=45 Jul 09 04:34:57.663089 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.663058 2571 generic.go:356] "Generic (PLEG): container finished" podID="579df4c9-299a-400b-b783-a7999380d127" containerID="edfc13286c263c6536befcb97cc95e044c7813c0d4adb9fe86eca4bf3c621a4a" exitCode=0 Jul 09 04:34:57.663205 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.663107 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" Jul 09 04:34:57.663205 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.663119 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" event={"ID":"579df4c9-299a-400b-b783-a7999380d127","Type":"ContainerDied","Data":"edfc13286c263c6536befcb97cc95e044c7813c0d4adb9fe86eca4bf3c621a4a"} Jul 09 04:34:57.663205 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.663143 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p" event={"ID":"579df4c9-299a-400b-b783-a7999380d127","Type":"ContainerDied","Data":"29d7a833e0fbff7411d59a19712b77d81de6d7ee8e9423ef7f82a37b311971cb"} Jul 09 04:34:57.663205 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.663158 2571 scope.go:117] "RemoveContainer" containerID="edfc13286c263c6536befcb97cc95e044c7813c0d4adb9fe86eca4bf3c621a4a" Jul 09 04:34:57.672247 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.672229 2571 scope.go:117] "RemoveContainer" containerID="edfc13286c263c6536befcb97cc95e044c7813c0d4adb9fe86eca4bf3c621a4a" Jul 09 04:34:57.672504 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:34:57.672487 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edfc13286c263c6536befcb97cc95e044c7813c0d4adb9fe86eca4bf3c621a4a\": container with ID starting with edfc13286c263c6536befcb97cc95e044c7813c0d4adb9fe86eca4bf3c621a4a not found: ID does not exist" containerID="edfc13286c263c6536befcb97cc95e044c7813c0d4adb9fe86eca4bf3c621a4a" Jul 09 04:34:57.672591 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.672512 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edfc13286c263c6536befcb97cc95e044c7813c0d4adb9fe86eca4bf3c621a4a"} err="failed to get container status \"edfc13286c263c6536befcb97cc95e044c7813c0d4adb9fe86eca4bf3c621a4a\": rpc error: code = NotFound desc = could not find container \"edfc13286c263c6536befcb97cc95e044c7813c0d4adb9fe86eca4bf3c621a4a\": container with ID starting with edfc13286c263c6536befcb97cc95e044c7813c0d4adb9fe86eca4bf3c621a4a not found: ID does not exist" Jul 09 04:34:57.681379 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.681358 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p"] Jul 09 04:34:57.686905 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.686885 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/sub-test-model-6bd312-kserve-795d84d5d7-hl27p"] Jul 09 04:34:57.916422 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.916398 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:57.942615 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.942592 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eaefb4e-473f-41f5-a4cd-f6d32d196957-tls-certs\") pod \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " Jul 09 04:34:57.942759 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.942629 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf2tj\" (UniqueName: \"kubernetes.io/projected/9eaefb4e-473f-41f5-a4cd-f6d32d196957-kube-api-access-nf2tj\") pod \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " Jul 09 04:34:57.942759 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.942659 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-dshm\") pod \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " Jul 09 04:34:57.942759 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.942698 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-tmp-dir\") pod \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " Jul 09 04:34:57.942759 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.942739 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-home\") pod \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " Jul 09 04:34:57.943040 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.942827 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-model-cache\") pod \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\" (UID: \"9eaefb4e-473f-41f5-a4cd-f6d32d196957\") " Jul 09 04:34:57.943040 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.943014 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9eaefb4e-473f-41f5-a4cd-f6d32d196957" (UID: "9eaefb4e-473f-41f5-a4cd-f6d32d196957"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:34:57.943166 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.943147 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-tmp-dir\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:34:57.943237 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.943208 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-home" (OuterVolumeSpecName: "home") pod "9eaefb4e-473f-41f5-a4cd-f6d32d196957" (UID: "9eaefb4e-473f-41f5-a4cd-f6d32d196957"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:34:57.943295 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.943215 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-model-cache" (OuterVolumeSpecName: "model-cache") pod "9eaefb4e-473f-41f5-a4cd-f6d32d196957" (UID: "9eaefb4e-473f-41f5-a4cd-f6d32d196957"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:34:57.944742 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.944712 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-dshm" (OuterVolumeSpecName: "dshm") pod "9eaefb4e-473f-41f5-a4cd-f6d32d196957" (UID: "9eaefb4e-473f-41f5-a4cd-f6d32d196957"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:34:57.944742 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.944728 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eaefb4e-473f-41f5-a4cd-f6d32d196957-kube-api-access-nf2tj" (OuterVolumeSpecName: "kube-api-access-nf2tj") pod "9eaefb4e-473f-41f5-a4cd-f6d32d196957" (UID: "9eaefb4e-473f-41f5-a4cd-f6d32d196957"). InnerVolumeSpecName "kube-api-access-nf2tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:34:57.944912 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:57.944738 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eaefb4e-473f-41f5-a4cd-f6d32d196957-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9eaefb4e-473f-41f5-a4cd-f6d32d196957" (UID: "9eaefb4e-473f-41f5-a4cd-f6d32d196957"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:34:58.044084 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:58.044059 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eaefb4e-473f-41f5-a4cd-f6d32d196957-tls-certs\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:34:58.044084 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:58.044082 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf2tj\" (UniqueName: \"kubernetes.io/projected/9eaefb4e-473f-41f5-a4cd-f6d32d196957-kube-api-access-nf2tj\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:34:58.044209 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:58.044091 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-dshm\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:34:58.044209 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:58.044099 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-home\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:34:58.044209 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:58.044107 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eaefb4e-473f-41f5-a4cd-f6d32d196957-model-cache\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:34:58.670886 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:58.670839 2571 generic.go:356] "Generic (PLEG): container finished" podID="9eaefb4e-473f-41f5-a4cd-f6d32d196957" containerID="f8f4cfdf21fc467404d8a51b04e51e1b7e573b3a53eace8f2c5699f2bf7baee6" exitCode=0 Jul 09 04:34:58.671050 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:58.670896 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" Jul 09 04:34:58.671050 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:58.670914 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" event={"ID":"9eaefb4e-473f-41f5-a4cd-f6d32d196957","Type":"ContainerDied","Data":"f8f4cfdf21fc467404d8a51b04e51e1b7e573b3a53eace8f2c5699f2bf7baee6"} Jul 09 04:34:58.671050 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:58.670942 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8" event={"ID":"9eaefb4e-473f-41f5-a4cd-f6d32d196957","Type":"ContainerDied","Data":"56c283a2af6f2d5542c7d482dee6a6c0440b0e4a5faa1044c7143e78f3ac4974"} Jul 09 04:34:58.671050 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:58.670966 2571 scope.go:117] "RemoveContainer" containerID="f8f4cfdf21fc467404d8a51b04e51e1b7e573b3a53eace8f2c5699f2bf7baee6" Jul 09 04:34:58.680251 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:58.680230 2571 scope.go:117] "RemoveContainer" containerID="f8f4cfdf21fc467404d8a51b04e51e1b7e573b3a53eace8f2c5699f2bf7baee6" Jul 09 04:34:58.680497 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:34:58.680479 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f4cfdf21fc467404d8a51b04e51e1b7e573b3a53eace8f2c5699f2bf7baee6\": container with ID starting with f8f4cfdf21fc467404d8a51b04e51e1b7e573b3a53eace8f2c5699f2bf7baee6 not found: ID does not exist" containerID="f8f4cfdf21fc467404d8a51b04e51e1b7e573b3a53eace8f2c5699f2bf7baee6" Jul 09 04:34:58.680567 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:58.680504 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f4cfdf21fc467404d8a51b04e51e1b7e573b3a53eace8f2c5699f2bf7baee6"} err="failed to get container status \"f8f4cfdf21fc467404d8a51b04e51e1b7e573b3a53eace8f2c5699f2bf7baee6\": rpc error: code = NotFound desc = could not find container \"f8f4cfdf21fc467404d8a51b04e51e1b7e573b3a53eace8f2c5699f2bf7baee6\": container with ID starting with f8f4cfdf21fc467404d8a51b04e51e1b7e573b3a53eace8f2c5699f2bf7baee6 not found: ID does not exist" Jul 09 04:34:58.693505 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:58.693485 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8"] Jul 09 04:34:58.699453 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:58.699430 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-589b59/sub-test-model-589b59-kserve-6ff5f69d4c-9swx8"] Jul 09 04:34:59.627464 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:59.627437 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="579df4c9-299a-400b-b783-a7999380d127" path="/var/lib/kubelet/pods/579df4c9-299a-400b-b783-a7999380d127/volumes" Jul 09 04:34:59.627804 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:34:59.627788 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eaefb4e-473f-41f5-a4cd-f6d32d196957" path="/var/lib/kubelet/pods/9eaefb4e-473f-41f5-a4cd-f6d32d196957/volumes" Jul 09 04:35:00.550903 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.550875 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2"] Jul 09 04:35:00.551212 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.551201 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eaefb4e-473f-41f5-a4cd-f6d32d196957" containerName="main" Jul 09 04:35:00.551254 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.551214 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eaefb4e-473f-41f5-a4cd-f6d32d196957" containerName="main" Jul 09 04:35:00.551254 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.551227 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="579df4c9-299a-400b-b783-a7999380d127" containerName="main" Jul 09 04:35:00.551254 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.551233 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="579df4c9-299a-400b-b783-a7999380d127" containerName="main" Jul 09 04:35:00.551349 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.551291 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eaefb4e-473f-41f5-a4cd-f6d32d196957" containerName="main" Jul 09 04:35:00.551349 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.551303 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="579df4c9-299a-400b-b783-a7999380d127" containerName="main" Jul 09 04:35:00.555556 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.555538 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.559105 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.559088 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6bd312\"/\"default-dockercfg-jdwn2\"" Jul 09 04:35:00.559217 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.559198 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6bd312\"/\"openshift-service-ca.crt\"" Jul 09 04:35:00.559217 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.559207 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6bd312\"/\"kube-root-ca.crt\"" Jul 09 04:35:00.559313 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.559229 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6bd312\"/\"rate-test-model-6bd312-kserve-self-signed-certs\"" Jul 09 04:35:00.565941 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.565921 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2"] Jul 09 04:35:00.662658 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.662635 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-tmp-dir\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.663007 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.662669 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-tls-certs\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.663007 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.662695 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-dshm\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.663007 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.662718 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klvnl\" (UniqueName: \"kubernetes.io/projected/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-kube-api-access-klvnl\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.663007 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.662808 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-home\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.663007 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.662825 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-model-cache\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.763565 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.763537 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-home\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.763565 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.763567 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-model-cache\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.763753 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.763598 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-tmp-dir\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.763753 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.763619 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-tls-certs\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.763753 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.763636 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-dshm\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.763753 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.763652 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klvnl\" (UniqueName: \"kubernetes.io/projected/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-kube-api-access-klvnl\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.763989 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.763952 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-model-cache\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.764045 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.763993 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-home\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.764045 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.764025 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-tmp-dir\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.765831 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.765806 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-dshm\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.765945 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.765935 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-tls-certs\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.771981 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.771961 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klvnl\" (UniqueName: \"kubernetes.io/projected/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-kube-api-access-klvnl\") pod \"rate-test-model-6bd312-kserve-757655c8cf-4gbt2\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.864548 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.864524 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:00.993493 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:00.993468 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2"] Jul 09 04:35:00.996175 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:35:00.996146 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26de8534_cca7_4ff8_8ef5_7a3b05b60a55.slice/crio-b869d4b9136586efdcfde5f51080113702ce30dd8c7aa23fbefa8dc42823e520 WatchSource:0}: Error finding container b869d4b9136586efdcfde5f51080113702ce30dd8c7aa23fbefa8dc42823e520: Status 404 returned error can't find the container with id b869d4b9136586efdcfde5f51080113702ce30dd8c7aa23fbefa8dc42823e520 Jul 09 04:35:01.691380 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:01.691338 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" event={"ID":"26de8534-cca7-4ff8-8ef5-7a3b05b60a55","Type":"ContainerStarted","Data":"9225f984b53bfb15f40620009ac782a83656293ea08d75500ef914c8abc7278a"} Jul 09 04:35:01.691809 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:01.691386 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" event={"ID":"26de8534-cca7-4ff8-8ef5-7a3b05b60a55","Type":"ContainerStarted","Data":"b869d4b9136586efdcfde5f51080113702ce30dd8c7aa23fbefa8dc42823e520"} Jul 09 04:35:01.708830 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:01.708788 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" podStartSLOduration=1.708775846 podStartE2EDuration="1.708775846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:35:01.706616718 +0000 UTC m=+2336.565487814" watchObservedRunningTime="2026-07-09 04:35:01.708775846 +0000 UTC m=+2336.567646966" Jul 09 04:35:06.955678 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:06.955647 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:35:10.865395 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:10.865367 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:10.865764 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:10.865418 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:10.877264 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:10.877243 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:11.750273 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:11.750247 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:35:42.641133 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.641104 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l"] Jul 09 04:35:42.644365 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.644349 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.647604 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.647573 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-589b59\"/\"rate-test-model-589b59-kserve-self-signed-certs\"" Jul 09 04:35:42.647736 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.647621 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-589b59\"/\"openshift-service-ca.crt\"" Jul 09 04:35:42.647736 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.647628 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-589b59\"/\"default-dockercfg-wh6tv\"" Jul 09 04:35:42.647736 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.647648 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-589b59\"/\"kube-root-ca.crt\"" Jul 09 04:35:42.654942 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.654919 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l"] Jul 09 04:35:42.776168 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.776144 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-home\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.776283 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.776170 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-model-cache\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.776283 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.776221 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-dshm\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.776283 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.776264 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vftrz\" (UniqueName: \"kubernetes.io/projected/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-kube-api-access-vftrz\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.776388 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.776324 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-tmp-dir\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.776388 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.776362 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-tls-certs\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.877375 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.877356 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-home\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.877496 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.877379 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-model-cache\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.877496 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.877416 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-dshm\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.877496 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.877431 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vftrz\" (UniqueName: \"kubernetes.io/projected/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-kube-api-access-vftrz\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.877496 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.877446 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-tmp-dir\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.877496 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.877483 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-tls-certs\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.877802 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.877769 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-home\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.877889 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.877806 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-model-cache\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.877935 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.877892 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-tmp-dir\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.879482 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.879466 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-dshm\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.879880 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.879844 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-tls-certs\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.885147 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.885127 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vftrz\" (UniqueName: \"kubernetes.io/projected/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-kube-api-access-vftrz\") pod \"rate-test-model-589b59-kserve-8467875d7f-7mb9l\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:42.953778 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:42.953725 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:43.080110 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:43.080087 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l"] Jul 09 04:35:43.081338 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:35:43.081314 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda17cbca7_f2e1_4d8e_87f9_da3bafa10b17.slice/crio-09984d1b885b0142271db0887a6bdae611905708f974fce092808a4f6bb00de8 WatchSource:0}: Error finding container 09984d1b885b0142271db0887a6bdae611905708f974fce092808a4f6bb00de8: Status 404 returned error can't find the container with id 09984d1b885b0142271db0887a6bdae611905708f974fce092808a4f6bb00de8 Jul 09 04:35:43.902977 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:43.902946 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" event={"ID":"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17","Type":"ContainerStarted","Data":"7f7b3ef783f2c211217e6612821c88a79a888213b14f43674a9e9083be14eca5"} Jul 09 04:35:43.903325 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:43.902984 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" event={"ID":"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17","Type":"ContainerStarted","Data":"09984d1b885b0142271db0887a6bdae611905708f974fce092808a4f6bb00de8"} Jul 09 04:35:43.918820 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:43.918769 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" podStartSLOduration=1.9187456840000001 podStartE2EDuration="1.918745684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:35:43.918211019 +0000 UTC m=+2378.777082112" watchObservedRunningTime="2026-07-09 04:35:43.918745684 +0000 UTC m=+2378.777616786" Jul 09 04:35:48.038980 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:48.038945 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:35:52.954108 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:52.954074 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:52.954552 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:52.954124 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:52.966203 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:52.966180 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:35:53.961828 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:35:53.961803 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:36:01.335593 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:01.335555 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:36:05.778325 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:05.778300 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:36:05.787171 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:05.787146 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:36:18.240992 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:18.240960 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:36:24.068226 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:24.068191 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2"] Jul 09 04:36:25.061441 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:25.061404 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l"] Jul 09 04:36:25.961524 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:25.961492 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7616-140030239791920"] Jul 09 04:36:25.965004 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:25.964986 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7616-140030239791920" Jul 09 04:36:25.967017 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:25.966997 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-xtjnz\"" Jul 09 04:36:25.969327 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:25.969310 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7616-140030239791920"] Jul 09 04:36:26.011353 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:26.011333 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t98gr\" (UniqueName: \"kubernetes.io/projected/d84d17c8-db5d-4a59-8267-a819fa742cfe-kube-api-access-t98gr\") pod \"test-curl-7616-140030239791920\" (UID: \"d84d17c8-db5d-4a59-8267-a819fa742cfe\") " pod="openshift-ingress/test-curl-7616-140030239791920" Jul 09 04:36:26.112301 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:26.112277 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t98gr\" (UniqueName: \"kubernetes.io/projected/d84d17c8-db5d-4a59-8267-a819fa742cfe-kube-api-access-t98gr\") pod \"test-curl-7616-140030239791920\" (UID: \"d84d17c8-db5d-4a59-8267-a819fa742cfe\") " pod="openshift-ingress/test-curl-7616-140030239791920" Jul 09 04:36:26.119534 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:26.119516 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t98gr\" (UniqueName: \"kubernetes.io/projected/d84d17c8-db5d-4a59-8267-a819fa742cfe-kube-api-access-t98gr\") pod \"test-curl-7616-140030239791920\" (UID: \"d84d17c8-db5d-4a59-8267-a819fa742cfe\") " pod="openshift-ingress/test-curl-7616-140030239791920" Jul 09 04:36:26.274766 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:26.274701 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7616-140030239791920" Jul 09 04:36:26.405012 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:26.404987 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7616-140030239791920"] Jul 09 04:36:26.406478 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:36:26.406454 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd84d17c8_db5d_4a59_8267_a819fa742cfe.slice/crio-ad494f1f5de8c83f7d7d66c639d580f51f611802caa5224fb3763ac5eb2cca41 WatchSource:0}: Error finding container ad494f1f5de8c83f7d7d66c639d580f51f611802caa5224fb3763ac5eb2cca41: Status 404 returned error can't find the container with id ad494f1f5de8c83f7d7d66c639d580f51f611802caa5224fb3763ac5eb2cca41 Jul 09 04:36:26.407831 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:26.407810 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 04:36:27.121963 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:27.121886 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7616-140030239791920" event={"ID":"d84d17c8-db5d-4a59-8267-a819fa742cfe","Type":"ContainerStarted","Data":"ad494f1f5de8c83f7d7d66c639d580f51f611802caa5224fb3763ac5eb2cca41"} Jul 09 04:36:28.128703 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:28.128670 2571 generic.go:356] "Generic (PLEG): container finished" podID="d84d17c8-db5d-4a59-8267-a819fa742cfe" containerID="75dacf6c7933e0791e8cef1d45251dd01fd52f188505f6d5f145d48c039caf17" exitCode=0 Jul 09 04:36:28.129113 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:28.128719 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7616-140030239791920" event={"ID":"d84d17c8-db5d-4a59-8267-a819fa742cfe","Type":"ContainerDied","Data":"75dacf6c7933e0791e8cef1d45251dd01fd52f188505f6d5f145d48c039caf17"} Jul 09 04:36:29.270562 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:29.270538 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7616-140030239791920" Jul 09 04:36:29.318828 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:29.318801 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7616-140030239791920_d84d17c8-db5d-4a59-8267-a819fa742cfe/test-curl-7616-140030239791920/0.log" Jul 09 04:36:29.336044 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:29.336022 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t98gr\" (UniqueName: \"kubernetes.io/projected/d84d17c8-db5d-4a59-8267-a819fa742cfe-kube-api-access-t98gr\") pod \"d84d17c8-db5d-4a59-8267-a819fa742cfe\" (UID: \"d84d17c8-db5d-4a59-8267-a819fa742cfe\") " Jul 09 04:36:29.338102 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:29.338075 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84d17c8-db5d-4a59-8267-a819fa742cfe-kube-api-access-t98gr" (OuterVolumeSpecName: "kube-api-access-t98gr") pod "d84d17c8-db5d-4a59-8267-a819fa742cfe" (UID: "d84d17c8-db5d-4a59-8267-a819fa742cfe"). InnerVolumeSpecName "kube-api-access-t98gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:36:29.358263 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:29.358230 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7616-140030239791920"] Jul 09 04:36:29.360453 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:29.360432 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7616-140030239791920"] Jul 09 04:36:29.437615 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:29.437558 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t98gr\" (UniqueName: \"kubernetes.io/projected/d84d17c8-db5d-4a59-8267-a819fa742cfe-kube-api-access-t98gr\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:29.626301 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:29.626275 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d84d17c8-db5d-4a59-8267-a819fa742cfe" path="/var/lib/kubelet/pods/d84d17c8-db5d-4a59-8267-a819fa742cfe/volumes" Jul 09 04:36:30.140484 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:30.140458 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7616-140030239791920" Jul 09 04:36:30.140484 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:30.140448 2571 scope.go:117] "RemoveContainer" containerID="75dacf6c7933e0791e8cef1d45251dd01fd52f188505f6d5f145d48c039caf17" Jul 09 04:36:32.974421 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:32.974389 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7616-140030239894528"] Jul 09 04:36:32.974780 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:32.974723 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d84d17c8-db5d-4a59-8267-a819fa742cfe" containerName="test-curl-7616-140030239791920" Jul 09 04:36:32.974780 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:32.974735 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d84d17c8-db5d-4a59-8267-a819fa742cfe" containerName="test-curl-7616-140030239791920" Jul 09 04:36:32.974881 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:32.974814 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d84d17c8-db5d-4a59-8267-a819fa742cfe" containerName="test-curl-7616-140030239791920" Jul 09 04:36:32.979272 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:32.979248 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7616-140030239894528" Jul 09 04:36:32.981494 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:32.981469 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-xtjnz\"" Jul 09 04:36:32.981627 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:32.981519 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7616-140030239894528"] Jul 09 04:36:33.064136 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:33.064106 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9qn8\" (UniqueName: \"kubernetes.io/projected/ae3beaef-7f18-4583-996c-9d24c917296e-kube-api-access-b9qn8\") pod \"test-curl-7616-140030239894528\" (UID: \"ae3beaef-7f18-4583-996c-9d24c917296e\") " pod="openshift-ingress/test-curl-7616-140030239894528" Jul 09 04:36:33.164507 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:33.164485 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9qn8\" (UniqueName: \"kubernetes.io/projected/ae3beaef-7f18-4583-996c-9d24c917296e-kube-api-access-b9qn8\") pod \"test-curl-7616-140030239894528\" (UID: \"ae3beaef-7f18-4583-996c-9d24c917296e\") " pod="openshift-ingress/test-curl-7616-140030239894528" Jul 09 04:36:33.172535 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:33.172510 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9qn8\" (UniqueName: \"kubernetes.io/projected/ae3beaef-7f18-4583-996c-9d24c917296e-kube-api-access-b9qn8\") pod \"test-curl-7616-140030239894528\" (UID: \"ae3beaef-7f18-4583-996c-9d24c917296e\") " pod="openshift-ingress/test-curl-7616-140030239894528" Jul 09 04:36:33.288675 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:33.288625 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7616-140030239894528" Jul 09 04:36:33.414366 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:33.414340 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7616-140030239894528"] Jul 09 04:36:33.415636 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:36:33.415606 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae3beaef_7f18_4583_996c_9d24c917296e.slice/crio-0713fad1e48abb8d745bcf9ea68fe302fed946219bdd792c673b50c903a49728 WatchSource:0}: Error finding container 0713fad1e48abb8d745bcf9ea68fe302fed946219bdd792c673b50c903a49728: Status 404 returned error can't find the container with id 0713fad1e48abb8d745bcf9ea68fe302fed946219bdd792c673b50c903a49728 Jul 09 04:36:34.162102 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:34.162071 2571 generic.go:356] "Generic (PLEG): container finished" podID="ae3beaef-7f18-4583-996c-9d24c917296e" containerID="3bc719da1960a42492c7670bb260a1d7995ec63715b8787c8a99066c49a7d003" exitCode=0 Jul 09 04:36:34.162421 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:34.162156 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7616-140030239894528" event={"ID":"ae3beaef-7f18-4583-996c-9d24c917296e","Type":"ContainerDied","Data":"3bc719da1960a42492c7670bb260a1d7995ec63715b8787c8a99066c49a7d003"} Jul 09 04:36:34.162421 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:34.162195 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7616-140030239894528" event={"ID":"ae3beaef-7f18-4583-996c-9d24c917296e","Type":"ContainerStarted","Data":"0713fad1e48abb8d745bcf9ea68fe302fed946219bdd792c673b50c903a49728"} Jul 09 04:36:35.307843 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:35.307823 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7616-140030239894528" Jul 09 04:36:35.326169 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:35.326141 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7616-140030239894528_ae3beaef-7f18-4583-996c-9d24c917296e/test-curl-7616-140030239894528/0.log" Jul 09 04:36:35.349983 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:35.349956 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7616-140030239894528"] Jul 09 04:36:35.353345 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:35.353308 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7616-140030239894528"] Jul 09 04:36:35.384523 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:35.384504 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9qn8\" (UniqueName: \"kubernetes.io/projected/ae3beaef-7f18-4583-996c-9d24c917296e-kube-api-access-b9qn8\") pod \"ae3beaef-7f18-4583-996c-9d24c917296e\" (UID: \"ae3beaef-7f18-4583-996c-9d24c917296e\") " Jul 09 04:36:35.386254 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:35.386236 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3beaef-7f18-4583-996c-9d24c917296e-kube-api-access-b9qn8" (OuterVolumeSpecName: "kube-api-access-b9qn8") pod "ae3beaef-7f18-4583-996c-9d24c917296e" (UID: "ae3beaef-7f18-4583-996c-9d24c917296e"). InnerVolumeSpecName "kube-api-access-b9qn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:36:35.486098 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:35.486044 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9qn8\" (UniqueName: \"kubernetes.io/projected/ae3beaef-7f18-4583-996c-9d24c917296e-kube-api-access-b9qn8\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:35.627155 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:35.627130 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae3beaef-7f18-4583-996c-9d24c917296e" path="/var/lib/kubelet/pods/ae3beaef-7f18-4583-996c-9d24c917296e/volumes" Jul 09 04:36:36.173534 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:36.173506 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7616-140030239894528" Jul 09 04:36:36.173724 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:36.173512 2571 scope.go:117] "RemoveContainer" containerID="3bc719da1960a42492c7670bb260a1d7995ec63715b8787c8a99066c49a7d003" Jul 09 04:36:37.239418 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:37.239387 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:36:39.088638 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.088599 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" podUID="26de8534-cca7-4ff8-8ef5-7a3b05b60a55" containerName="main" containerID="cri-o://9225f984b53bfb15f40620009ac782a83656293ea08d75500ef914c8abc7278a" gracePeriod=45 Jul 09 04:36:39.344945 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.344918 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:36:39.421332 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.421297 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klvnl\" (UniqueName: \"kubernetes.io/projected/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-kube-api-access-klvnl\") pod \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " Jul 09 04:36:39.421531 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.421367 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-tmp-dir\") pod \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " Jul 09 04:36:39.421531 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.421401 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-model-cache\") pod \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " Jul 09 04:36:39.421531 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.421423 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-tls-certs\") pod \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " Jul 09 04:36:39.421531 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.421455 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-dshm\") pod \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " Jul 09 04:36:39.421531 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.421512 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-home\") pod \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\" (UID: \"26de8534-cca7-4ff8-8ef5-7a3b05b60a55\") " Jul 09 04:36:39.421795 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.421651 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26de8534-cca7-4ff8-8ef5-7a3b05b60a55" (UID: "26de8534-cca7-4ff8-8ef5-7a3b05b60a55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:36:39.421795 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.421669 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-model-cache" (OuterVolumeSpecName: "model-cache") pod "26de8534-cca7-4ff8-8ef5-7a3b05b60a55" (UID: "26de8534-cca7-4ff8-8ef5-7a3b05b60a55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:36:39.421929 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.421806 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-home" (OuterVolumeSpecName: "home") pod "26de8534-cca7-4ff8-8ef5-7a3b05b60a55" (UID: "26de8534-cca7-4ff8-8ef5-7a3b05b60a55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:36:39.421929 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.421899 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-tmp-dir\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:39.421929 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.421912 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-model-cache\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:39.421929 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.421924 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-home\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:39.423487 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.423455 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26de8534-cca7-4ff8-8ef5-7a3b05b60a55" (UID: "26de8534-cca7-4ff8-8ef5-7a3b05b60a55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:36:39.423597 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.423505 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-dshm" (OuterVolumeSpecName: "dshm") pod "26de8534-cca7-4ff8-8ef5-7a3b05b60a55" (UID: "26de8534-cca7-4ff8-8ef5-7a3b05b60a55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:36:39.423597 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.423530 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-kube-api-access-klvnl" (OuterVolumeSpecName: "kube-api-access-klvnl") pod "26de8534-cca7-4ff8-8ef5-7a3b05b60a55" (UID: "26de8534-cca7-4ff8-8ef5-7a3b05b60a55"). InnerVolumeSpecName "kube-api-access-klvnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:36:39.522389 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.522349 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klvnl\" (UniqueName: \"kubernetes.io/projected/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-kube-api-access-klvnl\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:39.522389 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.522380 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-tls-certs\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:39.522389 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:39.522390 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26de8534-cca7-4ff8-8ef5-7a3b05b60a55-dshm\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:40.080352 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.080239 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" podUID="a17cbca7-f2e1-4d8e-87f9-da3bafa10b17" containerName="main" containerID="cri-o://7f7b3ef783f2c211217e6612821c88a79a888213b14f43674a9e9083be14eca5" gracePeriod=45 Jul 09 04:36:40.198533 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.198496 2571 generic.go:356] "Generic (PLEG): container finished" podID="a17cbca7-f2e1-4d8e-87f9-da3bafa10b17" containerID="7f7b3ef783f2c211217e6612821c88a79a888213b14f43674a9e9083be14eca5" exitCode=0 Jul 09 04:36:40.199017 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.198527 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" event={"ID":"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17","Type":"ContainerDied","Data":"7f7b3ef783f2c211217e6612821c88a79a888213b14f43674a9e9083be14eca5"} Jul 09 04:36:40.199897 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.199872 2571 generic.go:356] "Generic (PLEG): container finished" podID="26de8534-cca7-4ff8-8ef5-7a3b05b60a55" containerID="9225f984b53bfb15f40620009ac782a83656293ea08d75500ef914c8abc7278a" exitCode=0 Jul 09 04:36:40.200032 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.199924 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" Jul 09 04:36:40.200032 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.199955 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" event={"ID":"26de8534-cca7-4ff8-8ef5-7a3b05b60a55","Type":"ContainerDied","Data":"9225f984b53bfb15f40620009ac782a83656293ea08d75500ef914c8abc7278a"} Jul 09 04:36:40.200032 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.200004 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2" event={"ID":"26de8534-cca7-4ff8-8ef5-7a3b05b60a55","Type":"ContainerDied","Data":"b869d4b9136586efdcfde5f51080113702ce30dd8c7aa23fbefa8dc42823e520"} Jul 09 04:36:40.200032 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.200024 2571 scope.go:117] "RemoveContainer" containerID="9225f984b53bfb15f40620009ac782a83656293ea08d75500ef914c8abc7278a" Jul 09 04:36:40.209158 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.209138 2571 scope.go:117] "RemoveContainer" containerID="9225f984b53bfb15f40620009ac782a83656293ea08d75500ef914c8abc7278a" Jul 09 04:36:40.209491 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:36:40.209473 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9225f984b53bfb15f40620009ac782a83656293ea08d75500ef914c8abc7278a\": container with ID starting with 9225f984b53bfb15f40620009ac782a83656293ea08d75500ef914c8abc7278a not found: ID does not exist" containerID="9225f984b53bfb15f40620009ac782a83656293ea08d75500ef914c8abc7278a" Jul 09 04:36:40.209561 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.209502 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9225f984b53bfb15f40620009ac782a83656293ea08d75500ef914c8abc7278a"} err="failed to get container status \"9225f984b53bfb15f40620009ac782a83656293ea08d75500ef914c8abc7278a\": rpc error: code = NotFound desc = could not find container \"9225f984b53bfb15f40620009ac782a83656293ea08d75500ef914c8abc7278a\": container with ID starting with 9225f984b53bfb15f40620009ac782a83656293ea08d75500ef914c8abc7278a not found: ID does not exist" Jul 09 04:36:40.216551 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.216521 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2"] Jul 09 04:36:40.220347 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.220320 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6bd312/rate-test-model-6bd312-kserve-757655c8cf-4gbt2"] Jul 09 04:36:40.335768 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.335699 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:36:40.430259 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.430225 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-home\") pod \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " Jul 09 04:36:40.430259 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.430267 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-model-cache\") pod \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " Jul 09 04:36:40.430463 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.430325 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-dshm\") pod \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " Jul 09 04:36:40.430463 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.430358 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vftrz\" (UniqueName: \"kubernetes.io/projected/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-kube-api-access-vftrz\") pod \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " Jul 09 04:36:40.430463 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.430405 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-tmp-dir\") pod \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " Jul 09 04:36:40.430463 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.430441 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-tls-certs\") pod \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\" (UID: \"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17\") " Jul 09 04:36:40.430671 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.430573 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-home" (OuterVolumeSpecName: "home") pod "a17cbca7-f2e1-4d8e-87f9-da3bafa10b17" (UID: "a17cbca7-f2e1-4d8e-87f9-da3bafa10b17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:36:40.430671 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.430568 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-model-cache" (OuterVolumeSpecName: "model-cache") pod "a17cbca7-f2e1-4d8e-87f9-da3bafa10b17" (UID: "a17cbca7-f2e1-4d8e-87f9-da3bafa10b17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:36:40.430781 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.430719 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a17cbca7-f2e1-4d8e-87f9-da3bafa10b17" (UID: "a17cbca7-f2e1-4d8e-87f9-da3bafa10b17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:36:40.430781 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.430739 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-home\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:40.430781 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.430772 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-model-cache\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:40.432423 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.432402 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-dshm" (OuterVolumeSpecName: "dshm") pod "a17cbca7-f2e1-4d8e-87f9-da3bafa10b17" (UID: "a17cbca7-f2e1-4d8e-87f9-da3bafa10b17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:36:40.432527 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.432515 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-kube-api-access-vftrz" (OuterVolumeSpecName: "kube-api-access-vftrz") pod "a17cbca7-f2e1-4d8e-87f9-da3bafa10b17" (UID: "a17cbca7-f2e1-4d8e-87f9-da3bafa10b17"). InnerVolumeSpecName "kube-api-access-vftrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:36:40.432589 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.432520 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a17cbca7-f2e1-4d8e-87f9-da3bafa10b17" (UID: "a17cbca7-f2e1-4d8e-87f9-da3bafa10b17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:36:40.531826 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.531794 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-tmp-dir\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:40.531826 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.531820 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-tls-certs\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:40.531826 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.531829 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-dshm\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:40.532068 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.531838 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vftrz\" (UniqueName: \"kubernetes.io/projected/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17-kube-api-access-vftrz\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:40.982172 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.982136 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7616-140030239895392"] Jul 09 04:36:40.982519 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.982505 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae3beaef-7f18-4583-996c-9d24c917296e" containerName="test-curl-7616-140030239894528" Jul 09 04:36:40.982581 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.982522 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae3beaef-7f18-4583-996c-9d24c917296e" containerName="test-curl-7616-140030239894528" Jul 09 04:36:40.982581 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.982537 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17cbca7-f2e1-4d8e-87f9-da3bafa10b17" containerName="main" Jul 09 04:36:40.982581 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.982543 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17cbca7-f2e1-4d8e-87f9-da3bafa10b17" containerName="main" Jul 09 04:36:40.982581 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.982558 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26de8534-cca7-4ff8-8ef5-7a3b05b60a55" containerName="main" Jul 09 04:36:40.982581 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.982563 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="26de8534-cca7-4ff8-8ef5-7a3b05b60a55" containerName="main" Jul 09 04:36:40.982730 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.982628 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a17cbca7-f2e1-4d8e-87f9-da3bafa10b17" containerName="main" Jul 09 04:36:40.982730 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.982636 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae3beaef-7f18-4583-996c-9d24c917296e" containerName="test-curl-7616-140030239894528" Jul 09 04:36:40.982730 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.982642 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="26de8534-cca7-4ff8-8ef5-7a3b05b60a55" containerName="main" Jul 09 04:36:40.985776 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.985760 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7616-140030239895392" Jul 09 04:36:40.987937 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.987911 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-xtjnz\"" Jul 09 04:36:40.991190 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:40.991168 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7616-140030239895392"] Jul 09 04:36:41.034880 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:41.034818 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpcbt\" (UniqueName: \"kubernetes.io/projected/44c65dc0-8c7d-446c-b5c3-0e58fa5255cd-kube-api-access-hpcbt\") pod \"test-curl-7616-140030239895392\" (UID: \"44c65dc0-8c7d-446c-b5c3-0e58fa5255cd\") " pod="openshift-ingress/test-curl-7616-140030239895392" Jul 09 04:36:41.135883 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:41.135818 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpcbt\" (UniqueName: \"kubernetes.io/projected/44c65dc0-8c7d-446c-b5c3-0e58fa5255cd-kube-api-access-hpcbt\") pod \"test-curl-7616-140030239895392\" (UID: \"44c65dc0-8c7d-446c-b5c3-0e58fa5255cd\") " pod="openshift-ingress/test-curl-7616-140030239895392" Jul 09 04:36:41.143598 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:41.143572 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpcbt\" (UniqueName: \"kubernetes.io/projected/44c65dc0-8c7d-446c-b5c3-0e58fa5255cd-kube-api-access-hpcbt\") pod \"test-curl-7616-140030239895392\" (UID: \"44c65dc0-8c7d-446c-b5c3-0e58fa5255cd\") " pod="openshift-ingress/test-curl-7616-140030239895392" Jul 09 04:36:41.206744 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:41.206708 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" Jul 09 04:36:41.207247 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:41.206710 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l" event={"ID":"a17cbca7-f2e1-4d8e-87f9-da3bafa10b17","Type":"ContainerDied","Data":"09984d1b885b0142271db0887a6bdae611905708f974fce092808a4f6bb00de8"} Jul 09 04:36:41.207247 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:41.206839 2571 scope.go:117] "RemoveContainer" containerID="7f7b3ef783f2c211217e6612821c88a79a888213b14f43674a9e9083be14eca5" Jul 09 04:36:41.228710 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:41.228679 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l"] Jul 09 04:36:41.231988 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:41.231959 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-589b59/rate-test-model-589b59-kserve-8467875d7f-7mb9l"] Jul 09 04:36:41.296128 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:41.296044 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7616-140030239895392" Jul 09 04:36:41.431894 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:41.431865 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7616-140030239895392"] Jul 09 04:36:41.433834 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:36:41.433804 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44c65dc0_8c7d_446c_b5c3_0e58fa5255cd.slice/crio-054248845c95b14a29d1c17c986353ef2bb8181b131c2aad110daa09ffb57be5 WatchSource:0}: Error finding container 054248845c95b14a29d1c17c986353ef2bb8181b131c2aad110daa09ffb57be5: Status 404 returned error can't find the container with id 054248845c95b14a29d1c17c986353ef2bb8181b131c2aad110daa09ffb57be5 Jul 09 04:36:41.626359 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:41.626329 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26de8534-cca7-4ff8-8ef5-7a3b05b60a55" path="/var/lib/kubelet/pods/26de8534-cca7-4ff8-8ef5-7a3b05b60a55/volumes" Jul 09 04:36:41.626791 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:41.626776 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a17cbca7-f2e1-4d8e-87f9-da3bafa10b17" path="/var/lib/kubelet/pods/a17cbca7-f2e1-4d8e-87f9-da3bafa10b17/volumes" Jul 09 04:36:42.215645 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:42.215612 2571 generic.go:356] "Generic (PLEG): container finished" podID="44c65dc0-8c7d-446c-b5c3-0e58fa5255cd" containerID="f2c429dc9dd183d9f39fcc67cbb16725faca4e9ea2be6931ae3c70202156647b" exitCode=0 Jul 09 04:36:42.216110 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:42.215701 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7616-140030239895392" event={"ID":"44c65dc0-8c7d-446c-b5c3-0e58fa5255cd","Type":"ContainerDied","Data":"f2c429dc9dd183d9f39fcc67cbb16725faca4e9ea2be6931ae3c70202156647b"} Jul 09 04:36:42.216110 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:42.215747 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7616-140030239895392" event={"ID":"44c65dc0-8c7d-446c-b5c3-0e58fa5255cd","Type":"ContainerStarted","Data":"054248845c95b14a29d1c17c986353ef2bb8181b131c2aad110daa09ffb57be5"} Jul 09 04:36:43.356478 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:43.356450 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7616-140030239895392" Jul 09 04:36:43.375502 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:43.375467 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7616-140030239895392_44c65dc0-8c7d-446c-b5c3-0e58fa5255cd/test-curl-7616-140030239895392/0.log" Jul 09 04:36:43.398045 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:43.398007 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7616-140030239895392"] Jul 09 04:36:43.400197 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:43.400171 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7616-140030239895392"] Jul 09 04:36:43.458291 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:43.458257 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpcbt\" (UniqueName: \"kubernetes.io/projected/44c65dc0-8c7d-446c-b5c3-0e58fa5255cd-kube-api-access-hpcbt\") pod \"44c65dc0-8c7d-446c-b5c3-0e58fa5255cd\" (UID: \"44c65dc0-8c7d-446c-b5c3-0e58fa5255cd\") " Jul 09 04:36:43.460347 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:43.460316 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c65dc0-8c7d-446c-b5c3-0e58fa5255cd-kube-api-access-hpcbt" (OuterVolumeSpecName: "kube-api-access-hpcbt") pod "44c65dc0-8c7d-446c-b5c3-0e58fa5255cd" (UID: "44c65dc0-8c7d-446c-b5c3-0e58fa5255cd"). InnerVolumeSpecName "kube-api-access-hpcbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:36:43.559308 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:43.559222 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpcbt\" (UniqueName: \"kubernetes.io/projected/44c65dc0-8c7d-446c-b5c3-0e58fa5255cd-kube-api-access-hpcbt\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:43.626145 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:43.626109 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44c65dc0-8c7d-446c-b5c3-0e58fa5255cd" path="/var/lib/kubelet/pods/44c65dc0-8c7d-446c-b5c3-0e58fa5255cd/volumes" Jul 09 04:36:44.227560 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:44.227531 2571 scope.go:117] "RemoveContainer" containerID="f2c429dc9dd183d9f39fcc67cbb16725faca4e9ea2be6931ae3c70202156647b" Jul 09 04:36:44.227721 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:44.227530 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7616-140030239895392" Jul 09 04:36:44.335938 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:44.335898 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7"] Jul 09 04:36:44.336164 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:44.336135 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" podUID="44820edc-8793-4ccd-a870-0435882ae126" containerName="maas-api" containerID="cri-o://26455a1f362f917a37e5dbc0e366dd19ae656dcac429a1976f721f57892dd6f0" gracePeriod=30 Jul 09 04:36:44.595315 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:44.595285 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" Jul 09 04:36:44.669548 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:44.669508 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44820edc-8793-4ccd-a870-0435882ae126-maas-api-tls\") pod \"44820edc-8793-4ccd-a870-0435882ae126\" (UID: \"44820edc-8793-4ccd-a870-0435882ae126\") " Jul 09 04:36:44.669548 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:44.669555 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9rb2\" (UniqueName: \"kubernetes.io/projected/44820edc-8793-4ccd-a870-0435882ae126-kube-api-access-b9rb2\") pod \"44820edc-8793-4ccd-a870-0435882ae126\" (UID: \"44820edc-8793-4ccd-a870-0435882ae126\") " Jul 09 04:36:44.669753 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:44.669604 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44820edc-8793-4ccd-a870-0435882ae126-openshift-service-ca\") pod \"44820edc-8793-4ccd-a870-0435882ae126\" (UID: \"44820edc-8793-4ccd-a870-0435882ae126\") " Jul 09 04:36:44.670087 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:44.670067 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44820edc-8793-4ccd-a870-0435882ae126-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "44820edc-8793-4ccd-a870-0435882ae126" (UID: "44820edc-8793-4ccd-a870-0435882ae126"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:36:44.671800 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:44.671756 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44820edc-8793-4ccd-a870-0435882ae126-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "44820edc-8793-4ccd-a870-0435882ae126" (UID: "44820edc-8793-4ccd-a870-0435882ae126"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:36:44.671920 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:44.671831 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44820edc-8793-4ccd-a870-0435882ae126-kube-api-access-b9rb2" (OuterVolumeSpecName: "kube-api-access-b9rb2") pod "44820edc-8793-4ccd-a870-0435882ae126" (UID: "44820edc-8793-4ccd-a870-0435882ae126"). InnerVolumeSpecName "kube-api-access-b9rb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:36:44.771188 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:44.771095 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44820edc-8793-4ccd-a870-0435882ae126-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:44.771188 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:44.771127 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44820edc-8793-4ccd-a870-0435882ae126-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:44.771188 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:44.771137 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9rb2\" (UniqueName: \"kubernetes.io/projected/44820edc-8793-4ccd-a870-0435882ae126-kube-api-access-b9rb2\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 09 04:36:45.235063 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:45.235031 2571 generic.go:356] "Generic (PLEG): container finished" podID="44820edc-8793-4ccd-a870-0435882ae126" containerID="26455a1f362f917a37e5dbc0e366dd19ae656dcac429a1976f721f57892dd6f0" exitCode=0 Jul 09 04:36:45.235246 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:45.235089 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" Jul 09 04:36:45.235246 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:45.235114 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" event={"ID":"44820edc-8793-4ccd-a870-0435882ae126","Type":"ContainerDied","Data":"26455a1f362f917a37e5dbc0e366dd19ae656dcac429a1976f721f57892dd6f0"} Jul 09 04:36:45.235246 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:45.235171 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7" event={"ID":"44820edc-8793-4ccd-a870-0435882ae126","Type":"ContainerDied","Data":"17adc32809e3ae3ba48c68a389812cc2e28ab5b55c501ffe0f96cac5d87641e9"} Jul 09 04:36:45.235246 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:45.235201 2571 scope.go:117] "RemoveContainer" containerID="26455a1f362f917a37e5dbc0e366dd19ae656dcac429a1976f721f57892dd6f0" Jul 09 04:36:45.244070 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:45.244049 2571 scope.go:117] "RemoveContainer" containerID="26455a1f362f917a37e5dbc0e366dd19ae656dcac429a1976f721f57892dd6f0" Jul 09 04:36:45.244307 ip-10-0-130-55 kubenswrapper[2571]: E0709 04:36:45.244290 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26455a1f362f917a37e5dbc0e366dd19ae656dcac429a1976f721f57892dd6f0\": container with ID starting with 26455a1f362f917a37e5dbc0e366dd19ae656dcac429a1976f721f57892dd6f0 not found: ID does not exist" containerID="26455a1f362f917a37e5dbc0e366dd19ae656dcac429a1976f721f57892dd6f0" Jul 09 04:36:45.244354 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:45.244317 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26455a1f362f917a37e5dbc0e366dd19ae656dcac429a1976f721f57892dd6f0"} err="failed to get container status \"26455a1f362f917a37e5dbc0e366dd19ae656dcac429a1976f721f57892dd6f0\": rpc error: code = NotFound desc = could not find container \"26455a1f362f917a37e5dbc0e366dd19ae656dcac429a1976f721f57892dd6f0\": container with ID starting with 26455a1f362f917a37e5dbc0e366dd19ae656dcac429a1976f721f57892dd6f0 not found: ID does not exist" Jul 09 04:36:45.255793 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:45.255764 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7"] Jul 09 04:36:45.259550 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:45.259526 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6bd312-5fb578976c-lc9l7"] Jul 09 04:36:45.626385 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:45.626349 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44820edc-8793-4ccd-a870-0435882ae126" path="/var/lib/kubelet/pods/44820edc-8793-4ccd-a870-0435882ae126/volumes" Jul 09 04:36:48.031525 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:36:48.031492 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:37:01.432615 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:01.432582 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m6wqq"] Jul 09 04:37:03.392732 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:03.392709 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d756d6f6f-dhdjz_1dc27941-2ef5-42f7-95cc-8915108a811b/copy-manifests/0.log" Jul 09 04:37:03.399508 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:03.399484 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d756d6f6f-dhdjz_1dc27941-2ef5-42f7-95cc-8915108a811b/manager/0.log" Jul 09 04:37:04.804551 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:04.804518 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2_85f28e4c-7cbb-4214-9095-e10b146b8c95/util/0.log" Jul 09 04:37:04.810809 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:04.810780 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2_85f28e4c-7cbb-4214-9095-e10b146b8c95/pull/0.log" Jul 09 04:37:04.816598 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:04.816579 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2_85f28e4c-7cbb-4214-9095-e10b146b8c95/extract/0.log" Jul 09 04:37:04.921756 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:04.921728 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4_82053c64-4844-42ae-83e3-fab214a4f537/extract/0.log" Jul 09 04:37:04.927966 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:04.927942 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4_82053c64-4844-42ae-83e3-fab214a4f537/util/0.log" Jul 09 04:37:04.934023 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:04.934005 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4_82053c64-4844-42ae-83e3-fab214a4f537/pull/0.log" Jul 09 04:37:05.036293 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:05.036263 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn_77ed061d-f09c-4e24-8b76-ed05fef5630f/util/0.log" Jul 09 04:37:05.042420 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:05.042383 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn_77ed061d-f09c-4e24-8b76-ed05fef5630f/pull/0.log" Jul 09 04:37:05.048195 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:05.048171 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn_77ed061d-f09c-4e24-8b76-ed05fef5630f/extract/0.log" Jul 09 04:37:05.153237 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:05.153211 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx_2924267d-1e47-42d4-8651-6fc55a0aaa22/util/0.log" Jul 09 04:37:05.159414 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:05.159393 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx_2924267d-1e47-42d4-8651-6fc55a0aaa22/pull/0.log" Jul 09 04:37:05.165594 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:05.165576 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx_2924267d-1e47-42d4-8651-6fc55a0aaa22/extract/0.log" Jul 09 04:37:05.925433 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:05.925411 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-m6wqq_3ac8490f-ff73-4272-8f68-782cfac6496f/limitador/0.log" Jul 09 04:37:06.699496 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:06.699466 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-644c76db6f-vttfs_d4b22e9d-d1ce-4539-aff3-e14c37347ba1/kube-auth-proxy/0.log" Jul 09 04:37:07.135094 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:07.135061 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7955c548df-lznzp_ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9/router/0.log" Jul 09 04:37:07.877792 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:07.877766 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-fbs28_8a80b4b0-6524-4def-9527-c7ece21d4497/main/0.log" Jul 09 04:37:07.986813 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:07.986789 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-9bjqc_dad24a98-995c-459b-b85d-7bf18ff5a3c8/main/0.log" Jul 09 04:37:11.480182 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.480145 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2mv72/must-gather-mvcbt"] Jul 09 04:37:11.480551 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.480476 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44c65dc0-8c7d-446c-b5c3-0e58fa5255cd" containerName="test-curl-7616-140030239895392" Jul 09 04:37:11.480551 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.480487 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="44c65dc0-8c7d-446c-b5c3-0e58fa5255cd" containerName="test-curl-7616-140030239895392" Jul 09 04:37:11.480551 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.480510 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44820edc-8793-4ccd-a870-0435882ae126" containerName="maas-api" Jul 09 04:37:11.480551 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.480516 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="44820edc-8793-4ccd-a870-0435882ae126" containerName="maas-api" Jul 09 04:37:11.480687 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.480577 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="44820edc-8793-4ccd-a870-0435882ae126" containerName="maas-api" Jul 09 04:37:11.480687 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.480584 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="44c65dc0-8c7d-446c-b5c3-0e58fa5255cd" containerName="test-curl-7616-140030239895392" Jul 09 04:37:11.485666 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.485642 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2mv72/must-gather-mvcbt" Jul 09 04:37:11.488169 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.488153 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2mv72\"/\"kube-root-ca.crt\"" Jul 09 04:37:11.488253 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.488153 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2mv72\"/\"openshift-service-ca.crt\"" Jul 09 04:37:11.488253 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.488153 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2mv72\"/\"default-dockercfg-cgqvf\"" Jul 09 04:37:11.493046 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.493028 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2mv72/must-gather-mvcbt"] Jul 09 04:37:11.502636 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.502616 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv2x5\" (UniqueName: \"kubernetes.io/projected/e0aa5d1f-8e80-4cc5-8db2-884d01a0d0c5-kube-api-access-bv2x5\") pod \"must-gather-mvcbt\" (UID: \"e0aa5d1f-8e80-4cc5-8db2-884d01a0d0c5\") " pod="openshift-must-gather-2mv72/must-gather-mvcbt" Jul 09 04:37:11.502750 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.502666 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0aa5d1f-8e80-4cc5-8db2-884d01a0d0c5-must-gather-output\") pod \"must-gather-mvcbt\" (UID: \"e0aa5d1f-8e80-4cc5-8db2-884d01a0d0c5\") " pod="openshift-must-gather-2mv72/must-gather-mvcbt" Jul 09 04:37:11.603320 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.603284 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2x5\" (UniqueName: \"kubernetes.io/projected/e0aa5d1f-8e80-4cc5-8db2-884d01a0d0c5-kube-api-access-bv2x5\") pod \"must-gather-mvcbt\" (UID: \"e0aa5d1f-8e80-4cc5-8db2-884d01a0d0c5\") " pod="openshift-must-gather-2mv72/must-gather-mvcbt" Jul 09 04:37:11.603485 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.603338 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0aa5d1f-8e80-4cc5-8db2-884d01a0d0c5-must-gather-output\") pod \"must-gather-mvcbt\" (UID: \"e0aa5d1f-8e80-4cc5-8db2-884d01a0d0c5\") " pod="openshift-must-gather-2mv72/must-gather-mvcbt" Jul 09 04:37:11.603630 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.603613 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0aa5d1f-8e80-4cc5-8db2-884d01a0d0c5-must-gather-output\") pod \"must-gather-mvcbt\" (UID: \"e0aa5d1f-8e80-4cc5-8db2-884d01a0d0c5\") " pod="openshift-must-gather-2mv72/must-gather-mvcbt" Jul 09 04:37:11.611635 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.611609 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv2x5\" (UniqueName: \"kubernetes.io/projected/e0aa5d1f-8e80-4cc5-8db2-884d01a0d0c5-kube-api-access-bv2x5\") pod \"must-gather-mvcbt\" (UID: \"e0aa5d1f-8e80-4cc5-8db2-884d01a0d0c5\") " pod="openshift-must-gather-2mv72/must-gather-mvcbt" Jul 09 04:37:11.796329 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.796219 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2mv72/must-gather-mvcbt" Jul 09 04:37:11.929713 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:11.929685 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2mv72/must-gather-mvcbt"] Jul 09 04:37:11.931824 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:37:11.931788 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0aa5d1f_8e80_4cc5_8db2_884d01a0d0c5.slice/crio-3d4a5878058742d13ad8904959d0e2dff3bbdafd645d5604e1671ed83df89b9e WatchSource:0}: Error finding container 3d4a5878058742d13ad8904959d0e2dff3bbdafd645d5604e1671ed83df89b9e: Status 404 returned error can't find the container with id 3d4a5878058742d13ad8904959d0e2dff3bbdafd645d5604e1671ed83df89b9e Jul 09 04:37:12.379784 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:12.379748 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2mv72/must-gather-mvcbt" event={"ID":"e0aa5d1f-8e80-4cc5-8db2-884d01a0d0c5","Type":"ContainerStarted","Data":"3d4a5878058742d13ad8904959d0e2dff3bbdafd645d5604e1671ed83df89b9e"} Jul 09 04:37:13.389670 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:13.389631 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2mv72/must-gather-mvcbt" event={"ID":"e0aa5d1f-8e80-4cc5-8db2-884d01a0d0c5","Type":"ContainerStarted","Data":"454125e9f106c264e41abfc74e8d38f93768e7af95ade6379373dab9334f7698"} Jul 09 04:37:13.390174 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:13.389686 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2mv72/must-gather-mvcbt" event={"ID":"e0aa5d1f-8e80-4cc5-8db2-884d01a0d0c5","Type":"ContainerStarted","Data":"23fa343357aea830368e265bcd4461a2b682471fc166b8afb7d0d73f901e7a17"} Jul 09 04:37:13.404803 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:13.404752 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2mv72/must-gather-mvcbt" podStartSLOduration=1.6061849540000002 podStartE2EDuration="2.404736617s" totalImagesPullingTime="798.551663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:37:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:37:13.403549638 +0000 UTC m=+2468.262420738" watchObservedRunningTime="2026-07-09 04:37:13.404736617 +0000 UTC m=+2468.263607717" Jul 09 04:37:14.214982 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:14.214957 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-28sj8_0d4983fe-8ee1-40b3-89f6-3508333126b3/global-pull-secret-syncer/0.log" Jul 09 04:37:14.387139 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:14.387108 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vf56w_388dfe09-6614-4173-a31b-65185f254dc4/konnectivity-agent/0.log" Jul 09 04:37:14.459579 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:14.459543 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-55.ec2.internal_77031bd827de1cac5285b54b00e4a311/haproxy/0.log" Jul 09 04:37:18.746021 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:18.745981 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2_85f28e4c-7cbb-4214-9095-e10b146b8c95/extract/0.log" Jul 09 04:37:18.765992 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:18.765968 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2_85f28e4c-7cbb-4214-9095-e10b146b8c95/util/0.log" Jul 09 04:37:18.787195 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:18.787140 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596k6t2_85f28e4c-7cbb-4214-9095-e10b146b8c95/pull/0.log" Jul 09 04:37:18.815873 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:18.815830 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4_82053c64-4844-42ae-83e3-fab214a4f537/extract/0.log" Jul 09 04:37:18.837278 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:18.837251 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4_82053c64-4844-42ae-83e3-fab214a4f537/util/0.log" Jul 09 04:37:18.858975 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:18.858945 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e026tv4_82053c64-4844-42ae-83e3-fab214a4f537/pull/0.log" Jul 09 04:37:18.898465 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:18.898436 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn_77ed061d-f09c-4e24-8b76-ed05fef5630f/extract/0.log" Jul 09 04:37:18.925257 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:18.925227 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn_77ed061d-f09c-4e24-8b76-ed05fef5630f/util/0.log" Jul 09 04:37:18.945440 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:18.945410 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jfssn_77ed061d-f09c-4e24-8b76-ed05fef5630f/pull/0.log" Jul 09 04:37:18.971878 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:18.971823 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx_2924267d-1e47-42d4-8651-6fc55a0aaa22/extract/0.log" Jul 09 04:37:18.992564 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:18.992531 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx_2924267d-1e47-42d4-8651-6fc55a0aaa22/util/0.log" Jul 09 04:37:19.013308 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:19.013228 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kpkfx_2924267d-1e47-42d4-8651-6fc55a0aaa22/pull/0.log" Jul 09 04:37:19.579144 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:19.579113 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-m6wqq_3ac8490f-ff73-4272-8f68-782cfac6496f/limitador/0.log" Jul 09 04:37:21.548992 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:21.548959 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-8ppvs_75a13467-5b99-4f2f-88fa-7fce616b4560/cluster-monitoring-operator/0.log" Jul 09 04:37:21.577508 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:21.577475 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-s6rnf_efdbdf52-eaeb-4c14-8218-6193304fa117/kube-state-metrics/0.log" Jul 09 04:37:21.604312 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:21.604280 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-s6rnf_efdbdf52-eaeb-4c14-8218-6193304fa117/kube-rbac-proxy-main/0.log" Jul 09 04:37:21.629534 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:21.629501 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-s6rnf_efdbdf52-eaeb-4c14-8218-6193304fa117/kube-rbac-proxy-self/0.log" Jul 09 04:37:21.873067 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:21.873045 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tkf4f_0139cc95-48ef-4dbc-adcc-8d24da56a352/node-exporter/0.log" Jul 09 04:37:21.893062 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:21.893034 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tkf4f_0139cc95-48ef-4dbc-adcc-8d24da56a352/kube-rbac-proxy/0.log" Jul 09 04:37:21.915357 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:21.915334 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tkf4f_0139cc95-48ef-4dbc-adcc-8d24da56a352/init-textfile/0.log" Jul 09 04:37:21.940702 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:21.940674 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-9wkz8_644e4ef9-8650-4747-950c-c3c02200e894/kube-rbac-proxy-main/0.log" Jul 09 04:37:21.969044 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:21.969005 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-9wkz8_644e4ef9-8650-4747-950c-c3c02200e894/kube-rbac-proxy-self/0.log" Jul 09 04:37:21.990384 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:21.990352 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-9wkz8_644e4ef9-8650-4747-950c-c3c02200e894/openshift-state-metrics/0.log" Jul 09 04:37:22.286125 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:22.286050 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58cfb49f5-dhcrh_7c22c7a1-ffe4-4d18-ba4c-7c78f408d889/telemeter-client/0.log" Jul 09 04:37:22.305934 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:22.305898 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58cfb49f5-dhcrh_7c22c7a1-ffe4-4d18-ba4c-7c78f408d889/reload/0.log" Jul 09 04:37:22.325741 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:22.325714 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58cfb49f5-dhcrh_7c22c7a1-ffe4-4d18-ba4c-7c78f408d889/kube-rbac-proxy/0.log" Jul 09 04:37:23.617874 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:23.617826 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-s486r_fc8d4547-833f-48d7-a30b-5c067c926f00/networking-console-plugin/0.log" Jul 09 04:37:24.195403 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.195363 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn"] Jul 09 04:37:24.205070 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.205036 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn"] Jul 09 04:37:24.205251 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.205136 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.264882 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.263879 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/2.log" Jul 09 04:37:24.267885 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.267476 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ln9ht_0341b993-2f81-4a86-a32d-8c26da6b80bb/console-operator/3.log" Jul 09 04:37:24.319971 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.319936 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5125ec26-45a2-4ccf-9a44-5acbd9edcf29-sys\") pod \"perf-node-gather-daemonset-sxntn\" (UID: \"5125ec26-45a2-4ccf-9a44-5acbd9edcf29\") " pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.320259 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.320230 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5125ec26-45a2-4ccf-9a44-5acbd9edcf29-lib-modules\") pod \"perf-node-gather-daemonset-sxntn\" (UID: \"5125ec26-45a2-4ccf-9a44-5acbd9edcf29\") " pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.320387 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.320281 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xcd8\" (UniqueName: \"kubernetes.io/projected/5125ec26-45a2-4ccf-9a44-5acbd9edcf29-kube-api-access-4xcd8\") pod \"perf-node-gather-daemonset-sxntn\" (UID: \"5125ec26-45a2-4ccf-9a44-5acbd9edcf29\") " pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.320477 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.320419 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5125ec26-45a2-4ccf-9a44-5acbd9edcf29-podres\") pod \"perf-node-gather-daemonset-sxntn\" (UID: \"5125ec26-45a2-4ccf-9a44-5acbd9edcf29\") " pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.320548 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.320524 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5125ec26-45a2-4ccf-9a44-5acbd9edcf29-proc\") pod \"perf-node-gather-daemonset-sxntn\" (UID: \"5125ec26-45a2-4ccf-9a44-5acbd9edcf29\") " pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.421284 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.421242 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5125ec26-45a2-4ccf-9a44-5acbd9edcf29-lib-modules\") pod \"perf-node-gather-daemonset-sxntn\" (UID: \"5125ec26-45a2-4ccf-9a44-5acbd9edcf29\") " pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.421472 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.421301 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xcd8\" (UniqueName: \"kubernetes.io/projected/5125ec26-45a2-4ccf-9a44-5acbd9edcf29-kube-api-access-4xcd8\") pod \"perf-node-gather-daemonset-sxntn\" (UID: \"5125ec26-45a2-4ccf-9a44-5acbd9edcf29\") " pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.421472 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.421339 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5125ec26-45a2-4ccf-9a44-5acbd9edcf29-podres\") pod \"perf-node-gather-daemonset-sxntn\" (UID: \"5125ec26-45a2-4ccf-9a44-5acbd9edcf29\") " pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.421472 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.421448 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5125ec26-45a2-4ccf-9a44-5acbd9edcf29-proc\") pod \"perf-node-gather-daemonset-sxntn\" (UID: \"5125ec26-45a2-4ccf-9a44-5acbd9edcf29\") " pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.421647 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.421472 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5125ec26-45a2-4ccf-9a44-5acbd9edcf29-podres\") pod \"perf-node-gather-daemonset-sxntn\" (UID: \"5125ec26-45a2-4ccf-9a44-5acbd9edcf29\") " pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.421647 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.421538 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5125ec26-45a2-4ccf-9a44-5acbd9edcf29-proc\") pod \"perf-node-gather-daemonset-sxntn\" (UID: \"5125ec26-45a2-4ccf-9a44-5acbd9edcf29\") " pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.421647 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.421565 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5125ec26-45a2-4ccf-9a44-5acbd9edcf29-sys\") pod \"perf-node-gather-daemonset-sxntn\" (UID: \"5125ec26-45a2-4ccf-9a44-5acbd9edcf29\") " pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.421647 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.421631 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5125ec26-45a2-4ccf-9a44-5acbd9edcf29-lib-modules\") pod \"perf-node-gather-daemonset-sxntn\" (UID: \"5125ec26-45a2-4ccf-9a44-5acbd9edcf29\") " pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.421647 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.421641 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5125ec26-45a2-4ccf-9a44-5acbd9edcf29-sys\") pod \"perf-node-gather-daemonset-sxntn\" (UID: \"5125ec26-45a2-4ccf-9a44-5acbd9edcf29\") " pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.429597 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.429569 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xcd8\" (UniqueName: \"kubernetes.io/projected/5125ec26-45a2-4ccf-9a44-5acbd9edcf29-kube-api-access-4xcd8\") pod \"perf-node-gather-daemonset-sxntn\" (UID: \"5125ec26-45a2-4ccf-9a44-5acbd9edcf29\") " pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.522834 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.522745 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:24.737052 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.737026 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn"] Jul 09 04:37:24.738388 ip-10-0-130-55 kubenswrapper[2571]: W0709 04:37:24.738362 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5125ec26_45a2_4ccf_9a44_5acbd9edcf29.slice/crio-e5579a63cf6661808b9172c8ea6f704528595118b3a79b912131cd50fc5e1497 WatchSource:0}: Error finding container e5579a63cf6661808b9172c8ea6f704528595118b3a79b912131cd50fc5e1497: Status 404 returned error can't find the container with id e5579a63cf6661808b9172c8ea6f704528595118b3a79b912131cd50fc5e1497 Jul 09 04:37:24.945325 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.945293 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d8fb6d944-82rp6_9725f88a-19c4-4203-aa62-2b33ea5feefd/console/0.log" Jul 09 04:37:24.981122 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:24.981085 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-npd5x_60ef45cd-61f2-41fe-b2a0-778e37e50563/download-server/0.log" Jul 09 04:37:25.480155 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:25.480125 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" event={"ID":"5125ec26-45a2-4ccf-9a44-5acbd9edcf29","Type":"ContainerStarted","Data":"9d69c7af93a835696282748cec91aeec8acf71568a1ae7ff42e1c916b78f83d4"} Jul 09 04:37:25.480342 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:25.480187 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" event={"ID":"5125ec26-45a2-4ccf-9a44-5acbd9edcf29","Type":"ContainerStarted","Data":"e5579a63cf6661808b9172c8ea6f704528595118b3a79b912131cd50fc5e1497"} Jul 09 04:37:25.480342 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:25.480221 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:25.496522 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:25.496477 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" podStartSLOduration=1.496459411 podStartE2EDuration="1.496459411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:37:25.494070649 +0000 UTC m=+2480.352941748" watchObservedRunningTime="2026-07-09 04:37:25.496459411 +0000 UTC m=+2480.355330508" Jul 09 04:37:25.544978 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:25.544946 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-c6pgx_aa89f5c0-4598-4f98-b76c-0748c8a8920b/volume-data-source-validator/0.log" Jul 09 04:37:26.578083 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:26.578048 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bcrtc_977c6161-7495-4f0d-96de-9a13fe980f4c/dns/0.log" Jul 09 04:37:26.597489 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:26.597450 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bcrtc_977c6161-7495-4f0d-96de-9a13fe980f4c/kube-rbac-proxy/0.log" Jul 09 04:37:26.621673 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:26.621639 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ltzfl_e23a13f6-edd0-42f2-a03e-ba0b1cf7186d/dns-node-resolver/0.log" Jul 09 04:37:27.203978 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:27.203951 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gn78p_15fb2067-4603-4c89-95d2-92f7f719166a/node-ca/0.log" Jul 09 04:37:28.367300 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:28.367269 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-644c76db6f-vttfs_d4b22e9d-d1ce-4539-aff3-e14c37347ba1/kube-auth-proxy/0.log" Jul 09 04:37:28.553188 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:28.553162 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7955c548df-lznzp_ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9/router/0.log" Jul 09 04:37:29.247634 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:29.247600 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j9hq8_f2b4080d-3736-4237-9b6f-40b058b4a35c/serve-healthcheck-canary/0.log" Jul 09 04:37:29.712237 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:29.710331 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-cv9wx_d38cb655-3e43-4c49-8d67-f750a00ea565/insights-operator/0.log" Jul 09 04:37:29.712742 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:29.712721 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-cv9wx_d38cb655-3e43-4c49-8d67-f750a00ea565/insights-operator/1.log" Jul 09 04:37:29.800823 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:29.800791 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-69dwx_266cb396-9250-4419-90ac-2609c2404209/kube-rbac-proxy/0.log" Jul 09 04:37:29.822394 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:29.822371 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-69dwx_266cb396-9250-4419-90ac-2609c2404209/exporter/0.log" Jul 09 04:37:29.844181 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:29.844143 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-69dwx_266cb396-9250-4419-90ac-2609c2404209/extractor/0.log" Jul 09 04:37:31.497581 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:31.497547 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2mv72/perf-node-gather-daemonset-sxntn" Jul 09 04:37:32.351695 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:32.351658 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d756d6f6f-dhdjz_1dc27941-2ef5-42f7-95cc-8915108a811b/manager/0.log" Jul 09 04:37:32.372441 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:32.372415 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d756d6f6f-dhdjz_1dc27941-2ef5-42f7-95cc-8915108a811b/copy-manifests/0.log" Jul 09 04:37:33.944815 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:33.944718 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-789458d7cb-x5n6f_d647fa53-001a-40c1-8440-fe68a50e9903/manager/0.log" Jul 09 04:37:34.688264 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:34.688227 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-644c76db6f-vttfs_d4b22e9d-d1ce-4539-aff3-e14c37347ba1/kube-auth-proxy/0.log" Jul 09 04:37:34.848701 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:34.848672 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7955c548df-lznzp_ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9/router/0.log" Jul 09 04:37:36.259924 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:36.259893 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-644c76db6f-vttfs_d4b22e9d-d1ce-4539-aff3-e14c37347ba1/kube-auth-proxy/0.log" Jul 09 04:37:36.425570 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:36.425537 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7955c548df-lznzp_ae76e9a1-9bf1-44a6-8e53-f94b66f74ef9/router/0.log" Jul 09 04:37:39.161441 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:39.161410 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-nsjvn_3ec70533-29e6-48ec-967a-e958f1926405/kube-storage-version-migrator-operator/1.log" Jul 09 04:37:39.163071 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:39.163035 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-nsjvn_3ec70533-29e6-48ec-967a-e958f1926405/kube-storage-version-migrator-operator/0.log" Jul 09 04:37:40.126545 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:40.126514 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-59wnp_a2217f5b-d7af-42e3-b058-b2e3570e8156/kube-multus/0.log" Jul 09 04:37:40.448907 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:40.448833 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bb7z2_747cde48-17af-4cda-ab13-0a621520741c/kube-multus-additional-cni-plugins/0.log" Jul 09 04:37:40.468442 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:40.468418 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bb7z2_747cde48-17af-4cda-ab13-0a621520741c/egress-router-binary-copy/0.log" Jul 09 04:37:40.490753 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:40.490734 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bb7z2_747cde48-17af-4cda-ab13-0a621520741c/cni-plugins/0.log" Jul 09 04:37:40.510521 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:40.510500 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bb7z2_747cde48-17af-4cda-ab13-0a621520741c/bond-cni-plugin/0.log" Jul 09 04:37:40.530668 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:40.530644 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bb7z2_747cde48-17af-4cda-ab13-0a621520741c/routeoverride-cni/0.log" Jul 09 04:37:40.551429 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:40.551409 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bb7z2_747cde48-17af-4cda-ab13-0a621520741c/whereabouts-cni-bincopy/0.log" Jul 09 04:37:40.571371 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:40.571347 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bb7z2_747cde48-17af-4cda-ab13-0a621520741c/whereabouts-cni/0.log" Jul 09 04:37:40.720402 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:40.720334 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fvb66_579aea83-7020-4eb2-9cc9-eb6318890a3e/network-metrics-daemon/0.log" Jul 09 04:37:40.736772 ip-10-0-130-55 kubenswrapper[2571]: I0709 04:37:40.736737 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fvb66_579aea83-7020-4eb2-9cc9-eb6318890a3e/kube-rbac-proxy/0.log"