Jul 22 18:54:39.522234 ip-10-0-139-51 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 18:54:39.522244 ip-10-0-139-51 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 18:54:39.522252 ip-10-0-139-51 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 18:54:39.522468 ip-10-0-139-51 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 18:54:50.048661 ip-10-0-139-51 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 22 18:54:50.048677 ip-10-0-139-51 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 86967b597fb2450e9629babf0029d75d -- Jul 22 18:56:36.970777 ip-10-0-139-51 systemd[1]: Starting Kubernetes Kubelet... Jul 22 18:56:37.353739 ip-10-0-139-51 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 18:56:37.353739 ip-10-0-139-51 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 18:56:37.353739 ip-10-0-139-51 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 18:56:37.353739 ip-10-0-139-51 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 18:56:37.353739 ip-10-0-139-51 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 18:56:37.355483 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.355364 2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 18:56:37.363195 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363166 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 18:56:37.363195 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363190 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 18:56:37.363195 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363195 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 18:56:37.363195 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363198 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 18:56:37.363195 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363202 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363208 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363211 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363214 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363217 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363220 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363223 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363226 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363228 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363231 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363234 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363236 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363239 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363242 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363245 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363247 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363250 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363252 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363255 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 18:56:37.363419 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363258 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363261 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363263 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363266 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363269 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363271 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363273 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363276 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363282 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363285 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363288 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363290 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363293 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363295 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363298 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363302 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363305 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363308 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363310 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363313 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 18:56:37.363887 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363316 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363318 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363321 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363323 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363325 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363328 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363331 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363334 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363336 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363339 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363341 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363344 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363346 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363349 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363351 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363354 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363356 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363358 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363360 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363363 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 18:56:37.364370 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363366 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363368 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363370 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363373 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363375 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363378 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363380 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363383 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363386 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363390 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363394 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363397 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363399 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363401 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363404 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363406 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363409 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363411 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363416 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 18:56:37.364870 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363419 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363422 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363425 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363428 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363431 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363433 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363436 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363438 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363441 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363444 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363446 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363449 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363451 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363454 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363456 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363459 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.363461 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364214 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364222 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364226 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 18:56:37.365329 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364228 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364231 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364233 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364236 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364239 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364243 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364246 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364250 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364253 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364255 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364258 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364261 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364264 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364267 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364271 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364274 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364277 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364279 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 18:56:37.365833 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364282 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364286 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364289 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364292 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364296 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364299 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364302 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364304 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364307 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364310 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364312 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364316 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364319 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364321 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364324 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364326 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364329 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364331 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364333 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 18:56:37.366282 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364336 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364338 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364340 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364343 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364345 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364347 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364350 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364352 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364355 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364359 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364362 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364365 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364367 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364370 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364373 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364376 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364378 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364381 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364384 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364387 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 18:56:37.366771 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364391 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364394 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364397 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364400 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364403 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364406 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364408 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364411 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364413 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364416 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364420 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364422 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364425 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364428 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364430 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364433 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364435 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364438 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364440 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364443 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 18:56:37.367264 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364445 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364448 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364450 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364453 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364458 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364460 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364463 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364465 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364468 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364470 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364473 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364475 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364477 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364480 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364482 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364485 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364488 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364511 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364515 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364961 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 18:56:37.367789 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364967 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364970 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364973 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364976 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364978 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364981 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364984 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364986 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364989 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364992 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364994 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364997 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.364999 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365002 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365004 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365007 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365012 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365014 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365017 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365019 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 18:56:37.368259 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365022 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365024 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365027 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365029 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365031 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365034 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365036 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365039 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365041 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365044 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365046 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365049 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365051 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365054 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365056 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365058 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365061 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365064 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365067 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365069 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 18:56:37.368764 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365071 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365074 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365076 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365079 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365081 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365084 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365086 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365091 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365098 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365101 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365104 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365107 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365109 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365112 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365114 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365117 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365120 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365124 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 18:56:37.369279 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365126 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365129 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365131 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365134 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365137 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365140 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365143 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365146 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365149 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365151 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365154 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365156 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365159 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365161 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365163 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365166 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365168 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365171 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365173 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365176 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 18:56:37.369836 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365178 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365181 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365185 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365188 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365191 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365194 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365196 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365199 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365201 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365204 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365221 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365225 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365228 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365231 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365234 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365237 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365240 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365242 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365245 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.365248 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 18:56:37.370338 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365329 2583 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365339 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365357 2583 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365363 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365369 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365374 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365382 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365387 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365390 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365393 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365397 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365400 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365403 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365406 2583 flags.go:64] FLAG: --cgroup-root="" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365410 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365413 2583 flags.go:64] FLAG: --client-ca-file="" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365417 2583 flags.go:64] FLAG: --cloud-provider="external" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365420 2583 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365428 2583 flags.go:64] FLAG: --cluster-domain="" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365431 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365434 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365437 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365441 2583 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365444 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 18:56:37.370871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365447 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365452 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365455 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365458 2583 flags.go:64] FLAG: --contention-profiling="false" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365460 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365463 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365467 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365470 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365474 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365477 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365480 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365483 2583 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365485 2583 flags.go:64] FLAG: --enable-server="true" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365488 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365513 2583 flags.go:64] FLAG: --event-burst="100" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365516 2583 flags.go:64] FLAG: --event-qps="50" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365519 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365522 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365525 2583 flags.go:64] FLAG: --eviction-hard="" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365529 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365532 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365535 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365539 2583 flags.go:64] FLAG: --eviction-soft="" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365542 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365546 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 18:56:37.371450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365555 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365558 2583 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365560 2583 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365563 2583 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365566 2583 flags.go:64] FLAG: --feature-gates="" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365569 2583 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365572 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365575 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365579 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365582 2583 flags.go:64] FLAG: --healthz-port="10248" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365585 2583 flags.go:64] FLAG: --help="false" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365588 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365591 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365594 2583 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365597 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365600 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365603 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365606 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365609 2583 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365611 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365614 2583 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365617 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365620 2583 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365623 2583 flags.go:64] FLAG: --kube-reserved="" Jul 22 18:56:37.372320 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365625 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365628 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365631 2583 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365634 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365636 2583 flags.go:64] FLAG: --lock-file="" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365640 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365643 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365646 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365651 2583 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365659 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365663 2583 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365665 2583 flags.go:64] FLAG: --logging-format="text" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365668 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365671 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365674 2583 flags.go:64] FLAG: --manifest-url="" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365676 2583 flags.go:64] FLAG: --manifest-url-header="" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365683 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365686 2583 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365690 2583 flags.go:64] FLAG: --max-pods="110" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365692 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365695 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365698 2583 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365701 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365703 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365706 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 18:56:37.373382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365709 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365717 2583 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365720 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365722 2583 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365725 2583 flags.go:64] FLAG: --pod-cidr="" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365728 2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365733 2583 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365736 2583 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365739 2583 flags.go:64] FLAG: --pods-per-core="0" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365742 2583 flags.go:64] FLAG: --port="10250" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365745 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365747 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b1c53953b6377a0e" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365750 2583 flags.go:64] FLAG: --qos-reserved="" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365755 2583 flags.go:64] FLAG: --read-only-port="10255" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365757 2583 flags.go:64] FLAG: --register-node="true" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365760 2583 flags.go:64] FLAG: --register-with-taints="" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365764 2583 flags.go:64] FLAG: --registry-burst="10" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365770 2583 flags.go:64] FLAG: --registry-qps="5" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365773 2583 flags.go:64] FLAG: --reserved-cpus="" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365776 2583 flags.go:64] FLAG: --reserved-memory="" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365780 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365783 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365785 2583 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365789 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365792 2583 flags.go:64] FLAG: --runonce="false" Jul 22 18:56:37.374122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365794 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365797 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365800 2583 flags.go:64] FLAG: --seccomp-default="false" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365803 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365806 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365809 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365812 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365815 2583 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365827 2583 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365830 2583 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365833 2583 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365836 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365839 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365842 2583 flags.go:64] FLAG: --system-cgroups="" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365845 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365851 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365853 2583 flags.go:64] FLAG: --tls-cert-file="" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365856 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365861 2583 flags.go:64] FLAG: --tls-min-version="" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365863 2583 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365867 2583 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365871 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365874 2583 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365876 2583 flags.go:64] FLAG: --v="2" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365880 2583 flags.go:64] FLAG: --version="false" Jul 22 18:56:37.375103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365885 2583 flags.go:64] FLAG: --vmodule="" Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365889 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.365892 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366014 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366017 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366021 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366024 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366027 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366029 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366032 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366035 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366037 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366040 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366043 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366045 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366048 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366050 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366053 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366056 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366058 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 18:56:37.376343 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366061 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366063 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366066 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366069 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366071 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366074 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366076 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366087 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366090 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366093 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366095 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366098 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366100 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366108 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366111 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366114 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366116 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366120 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366122 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366125 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 18:56:37.377108 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366127 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366130 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366132 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366135 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366137 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366139 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366142 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366144 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366147 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366149 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366152 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366155 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366157 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366159 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366162 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366165 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366167 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366169 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366172 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366176 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 18:56:37.377953 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366178 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366181 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366183 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366187 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366190 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366193 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366196 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366199 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366201 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366206 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366210 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366213 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366217 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366220 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366222 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366225 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366228 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366231 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366234 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 18:56:37.378666 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366237 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366239 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366242 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366244 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366247 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366249 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366252 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366254 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366256 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366259 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366262 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366264 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366268 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366270 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366273 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366276 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366278 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366280 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366283 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366286 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 18:56:37.379268 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366290 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 18:56:37.379806 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366293 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 18:56:37.379806 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.366296 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 18:56:37.379806 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.366302 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 18:56:37.379806 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.374860 2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 18:56:37.379806 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.374888 2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 18:56:37.379806 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.374916 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 18:56:37.379806 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.374925 2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 18:56:37.379806 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375012 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 18:56:37.379806 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375020 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 18:56:37.379806 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375025 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 18:56:37.379806 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375030 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 18:56:37.379806 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375034 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 18:56:37.379806 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375038 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 18:56:37.379806 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375042 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 18:56:37.379806 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375047 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375051 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375055 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375059 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375063 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375068 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375072 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375077 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375081 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375085 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375089 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375094 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375098 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375102 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375105 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375109 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375113 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375117 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375121 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375125 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 18:56:37.380178 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375129 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375133 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375139 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375144 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375148 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375153 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375157 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375161 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375166 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375170 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375176 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375183 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375188 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375192 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375197 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375202 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375206 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375210 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375215 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375219 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 18:56:37.380739 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375223 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375227 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375232 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375237 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375241 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375245 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375249 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375253 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375258 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375262 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375266 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375270 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375275 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375279 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375284 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375289 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375293 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375297 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375302 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 18:56:37.381289 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375306 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375310 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375317 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375322 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375327 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375332 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375337 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375341 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375345 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375349 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375354 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375359 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375363 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375367 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375371 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375375 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375379 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375383 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375387 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 18:56:37.381871 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375391 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 18:56:37.382330 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375395 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 18:56:37.382330 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375399 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 18:56:37.382330 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375403 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 18:56:37.382330 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375407 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 18:56:37.382330 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375411 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 22 18:56:37.382330 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375416 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 18:56:37.382330 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375420 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 18:56:37.382330 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375428 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 18:56:37.382330 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375435 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 18:56:37.382330 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375441 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 18:56:37.382330 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375445 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 18:56:37.382330 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375450 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 18:56:37.382330 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375454 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 18:56:37.382330 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.375463 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 18:56:37.382330 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375663 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375672 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375676 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375681 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375685 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375688 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375693 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375696 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375699 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375703 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375707 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375711 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375716 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375720 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375724 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375728 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375733 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375737 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375741 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375745 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 18:56:37.382961 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375750 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375754 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375759 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375763 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375768 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375772 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375776 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375782 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375790 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375794 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375799 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375803 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375807 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375812 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375815 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375820 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375824 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375828 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375832 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375836 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 18:56:37.383785 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375841 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375845 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375849 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375853 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375857 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375861 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375865 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375870 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375874 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375878 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375882 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375886 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375891 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375895 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375899 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375903 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375907 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375911 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375916 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375920 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 18:56:37.384676 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375925 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375930 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375934 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375939 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375945 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375951 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375956 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375961 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375966 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375973 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375978 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375983 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375987 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375992 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.375997 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376001 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376006 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376010 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376015 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 18:56:37.385275 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376019 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376023 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376027 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376031 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376035 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376038 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376042 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376046 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376051 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376055 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376060 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376065 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376069 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376073 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376077 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376082 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376086 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376090 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376094 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 18:56:37.385934 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:37.376098 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 18:56:37.386460 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.376106 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 18:56:37.386460 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.376298 2583 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 18:56:37.386460 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.378957 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 18:56:37.386460 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.380323 2583 server.go:1019] "Starting client certificate rotation" Jul 22 18:56:37.386460 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.380428 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 18:56:37.386460 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.380465 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 18:56:37.386460 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.383047 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 18:56:37.388941 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.388920 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 18:56:37.399974 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.399948 2583 log.go:25] "Validated CRI v1 runtime API" Jul 22 18:56:37.402591 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.402574 2583 log.go:25] "Validated CRI v1 image API" Jul 22 18:56:37.403399 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.403382 2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 18:56:37.406029 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.406004 2583 fs.go:135] Filesystem UUIDs: map[0fa8a763-3b4d-4431-85a1-d54092a268a3:/dev/nvme0n1p3 53e4caa2-6b94-4b14-a90e-f9f4a8153b79:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 22 18:56:37.406122 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.406026 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 18:56:37.410407 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.410381 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 18:56:37.411178 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.411055 2583 manager.go:216] Machine: {Timestamp:2026-07-22 18:56:37.40976935 +0000 UTC m=+0.321933410 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097203 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24fb14601398025df6ae023a1f109d SystemUUID:ec24fb14-6013-9802-5df6-ae023a1f109d BootID:86967b59-7fb2-450e-9629-babf0029d75d Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1c:e3:59:f2:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1c:e3:59:f2:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:bf:e6:45:4a:09 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 18:56:37.411178 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.411173 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 18:56:37.411330 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.411296 2583 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 18:56:37.414101 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.414070 2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 18:56:37.414282 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.414104 2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-51.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 18:56:37.414897 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.414885 2583 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 18:56:37.414952 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.414905 2583 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 18:56:37.414952 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.414923 2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 18:56:37.415623 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.415610 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 18:56:37.416740 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.416728 2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 18:56:37.419803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.419789 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 22 18:56:37.419930 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.419919 2583 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 18:56:37.419984 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.419962 2583 kubelet.go:486] "Attempting to sync node with API server" Jul 22 18:56:37.420041 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.419983 2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 18:56:37.420041 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.420005 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 18:56:37.420041 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.420019 2583 kubelet.go:398] "Adding apiserver pod source" Jul 22 18:56:37.420041 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.420034 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 18:56:37.420436 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.420422 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 18:56:37.420519 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.420446 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 18:56:37.422560 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.422546 2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 18:56:37.424252 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.424239 2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 18:56:37.424298 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.424269 2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 18:56:37.425185 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.425165 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 18:56:37.425185 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.425181 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 18:56:37.425185 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.425188 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 18:56:37.425336 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.425194 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 18:56:37.425336 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.425202 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 18:56:37.425336 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.425208 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 18:56:37.425336 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.425216 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 18:56:37.425336 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.425220 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 18:56:37.425336 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.425226 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 18:56:37.425336 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.425232 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 18:56:37.425336 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.425240 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 18:56:37.425336 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.425245 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 18:56:37.425997 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.425986 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 18:56:37.425997 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.425996 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 18:56:37.429055 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.429042 2583 server.go:1268] "Started kubelet" Jul 22 18:56:37.429155 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.429116 2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 18:56:37.429201 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.429127 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 18:56:37.432726 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.432698 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 18:56:37.433406 ip-10-0-139-51 systemd[1]: Started Kubernetes Kubelet. Jul 22 18:56:37.433662 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.433519 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-51.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 18:56:37.434075 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.433797 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 18:56:37.434075 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.433877 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-51.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 18:56:37.434546 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.434440 2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 18:56:37.435801 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.435781 2583 server.go:310] "Adding debug handlers to kubelet server" Jul 22 18:56:37.439782 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.439762 2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 18:56:37.439852 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.438973 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-51.ec2.internal.18c4b19309d7f3f6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-51.ec2.internal,UID:ip-10-0-139-51.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-51.ec2.internal,},FirstTimestamp:2026-07-22 18:56:37.429015542 +0000 UTC m=+0.341179600,LastTimestamp:2026-07-22 18:56:37.429015542 +0000 UTC m=+0.341179600,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-51.ec2.internal,}" Jul 22 18:56:37.442698 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.442682 2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 18:56:37.442698 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.442693 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 18:56:37.442794 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.442767 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Jul 22 18:56:37.443294 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.443273 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 18:56:37.443374 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.443295 2583 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 18:56:37.443374 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.443311 2583 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 18:56:37.443511 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.443445 2583 reconstruct.go:97] "Volume reconstruction finished" Jul 22 18:56:37.443511 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.443454 2583 reconciler.go:29] "Reconciler: start to sync state" Jul 22 18:56:37.443626 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.443568 2583 factory.go:55] Registering systemd factory Jul 22 18:56:37.443626 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.443582 2583 factory.go:223] Registration of the systemd container factory successfully Jul 22 18:56:37.444002 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.443984 2583 factory.go:153] Registering CRI-O factory Jul 22 18:56:37.444002 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.444001 2583 factory.go:223] Registration of the crio container factory successfully Jul 22 18:56:37.444106 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.444050 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 18:56:37.444106 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.444069 2583 factory.go:103] Registering Raw factory Jul 22 18:56:37.444106 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.444080 2583 manager.go:1196] Started watching for new ooms in manager Jul 22 18:56:37.444371 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.444361 2583 manager.go:349] Starting recovery of all containers Jul 22 18:56:37.452722 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.452690 2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-51.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 18:56:37.452824 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.452730 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 18:56:37.454090 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.454074 2583 manager.go:354] Recovery completed Jul 22 18:56:37.455243 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.455222 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4sbth" Jul 22 18:56:37.458072 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.458055 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 18:56:37.460344 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.460328 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory" Jul 22 18:56:37.460395 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.460361 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 18:56:37.460395 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.460372 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID" Jul 22 18:56:37.460966 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.460954 2583 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 18:56:37.460966 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.460965 2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 18:56:37.461070 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.460980 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 22 18:56:37.461070 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.461006 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 18:56:37.462206 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.462191 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory" Jul 22 18:56:37.462258 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.462217 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 18:56:37.462258 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.462232 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID" Jul 22 18:56:37.462375 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.462363 2583 policy_none.go:49] "None policy: Start" Jul 22 18:56:37.462409 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.462381 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 18:56:37.462409 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.462391 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 18:56:37.462605 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.462589 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4sbth" Jul 22 18:56:37.463553 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.463541 2583 policy_none.go:47] "Start" Jul 22 18:56:37.503257 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.503098 2583 manager.go:337] "Starting Device Plugin manager" Jul 22 18:56:37.503375 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.503282 2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 18:56:37.503375 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.503297 2583 server.go:87] "Starting device plugin registration server" Jul 22 18:56:37.503585 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.503574 2583 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 18:56:37.503618 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.503587 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 18:56:37.503725 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.503708 2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 18:56:37.503822 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.503810 2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 18:56:37.503822 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.503823 2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 18:56:37.504397 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.504378 2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 18:56:37.504532 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.504427 2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-51.ec2.internal\" not found" Jul 22 18:56:37.542216 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.542185 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 18:56:37.543553 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.543537 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 18:56:37.543629 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.543574 2583 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 18:56:37.543629 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.543595 2583 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 18:56:37.543720 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.543625 2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 18:56:37.546127 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.546110 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 18:56:37.604286 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.604213 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 18:56:37.605098 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.605080 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory" Jul 22 18:56:37.605192 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.605119 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 18:56:37.605192 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.605130 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID" Jul 22 18:56:37.605192 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.605154 2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.614533 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.614509 2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.614627 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.614536 2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-51.ec2.internal\": node \"ip-10-0-139-51.ec2.internal\" not found" Jul 22 18:56:37.636337 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.636311 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Jul 22 18:56:37.644485 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.644455 2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal"] Jul 22 18:56:37.644612 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.644530 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 18:56:37.645333 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.645319 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory" Jul 22 18:56:37.645429 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.645352 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 18:56:37.645429 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.645366 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID" Jul 22 18:56:37.646688 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.646673 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 18:56:37.646820 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.646804 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.646869 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.646837 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 18:56:37.647340 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.647317 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory" Jul 22 18:56:37.647416 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.647350 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory" Jul 22 18:56:37.647416 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.647368 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 18:56:37.647416 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.647378 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID" Jul 22 18:56:37.647540 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.647351 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 18:56:37.647540 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.647437 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID" Jul 22 18:56:37.648588 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.648574 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.648638 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.648600 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 18:56:37.649311 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.649296 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory" Jul 22 18:56:37.649376 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.649322 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 18:56:37.649376 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.649333 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID" Jul 22 18:56:37.664031 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.664012 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-51.ec2.internal\" not found" node="ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.667769 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.667755 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-51.ec2.internal\" not found" node="ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.736928 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.736888 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Jul 22 18:56:37.745244 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.745225 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d89308905f53d9806708f28d8cb1d8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"9d89308905f53d9806708f28d8cb1d8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.745317 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.745249 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f73e47efca17a7bdd186cdd86897ddfa-config\") pod \"kube-apiserver-proxy-ip-10-0-139-51.ec2.internal\" (UID: \"f73e47efca17a7bdd186cdd86897ddfa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.745317 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.745267 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d89308905f53d9806708f28d8cb1d8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"9d89308905f53d9806708f28d8cb1d8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.837437 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.837385 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Jul 22 18:56:37.845769 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.845738 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d89308905f53d9806708f28d8cb1d8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"9d89308905f53d9806708f28d8cb1d8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.845836 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.845775 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f73e47efca17a7bdd186cdd86897ddfa-config\") pod \"kube-apiserver-proxy-ip-10-0-139-51.ec2.internal\" (UID: \"f73e47efca17a7bdd186cdd86897ddfa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.845836 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.845826 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d89308905f53d9806708f28d8cb1d8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"9d89308905f53d9806708f28d8cb1d8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.845947 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.845843 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d89308905f53d9806708f28d8cb1d8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"9d89308905f53d9806708f28d8cb1d8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.845947 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.845870 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f73e47efca17a7bdd186cdd86897ddfa-config\") pod \"kube-apiserver-proxy-ip-10-0-139-51.ec2.internal\" (UID: \"f73e47efca17a7bdd186cdd86897ddfa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.845947 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.845843 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d89308905f53d9806708f28d8cb1d8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"9d89308905f53d9806708f28d8cb1d8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.938286 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:37.938199 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Jul 22 18:56:37.967743 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.967715 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Jul 22 18:56:37.970056 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:37.970034 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" Jul 22 18:56:38.039156 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:38.039103 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Jul 22 18:56:38.139701 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:38.139652 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Jul 22 18:56:38.245645 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:38.245555 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Jul 22 18:56:38.346332 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:38.346288 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Jul 22 18:56:38.379914 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:38.379892 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 18:56:38.380414 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:38.380039 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 18:56:38.446451 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:38.446424 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Jul 22 18:56:38.448693 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:38.448674 2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 18:56:38.465935 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:38.465899 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 18:51:38 +0000 UTC" deadline="2028-03-31 01:45:34.2653745 +0000 UTC" Jul 22 18:56:38.465935 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:38.465929 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14814h48m55.799449107s" Jul 22 18:56:38.469811 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:38.469784 2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 18:56:38.480560 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:38.480525 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d89308905f53d9806708f28d8cb1d8c.slice/crio-7b51e07599e0d8c1461cb856710a76e7b210c6ef83b1c04b8b16d66ca4d6d85c WatchSource:0}: Error finding container 7b51e07599e0d8c1461cb856710a76e7b210c6ef83b1c04b8b16d66ca4d6d85c: Status 404 returned error can't find the container with id 7b51e07599e0d8c1461cb856710a76e7b210c6ef83b1c04b8b16d66ca4d6d85c Jul 22 18:56:38.480816 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:38.480797 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73e47efca17a7bdd186cdd86897ddfa.slice/crio-3ae6e2eec063fb1f323edfe1732b1dec98223d1bdefb0b1eb1f80ca2bb772a87 WatchSource:0}: Error finding container 3ae6e2eec063fb1f323edfe1732b1dec98223d1bdefb0b1eb1f80ca2bb772a87: Status 404 returned error can't find the container with id 3ae6e2eec063fb1f323edfe1732b1dec98223d1bdefb0b1eb1f80ca2bb772a87 Jul 22 18:56:38.485557 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:38.485536 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 18:56:38.544422 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:38.544323 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" Jul 22 18:56:38.549832 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:38.549783 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" event={"ID":"9d89308905f53d9806708f28d8cb1d8c","Type":"ContainerStarted","Data":"7b51e07599e0d8c1461cb856710a76e7b210c6ef83b1c04b8b16d66ca4d6d85c"} Jul 22 18:56:38.550814 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:38.550791 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" event={"ID":"f73e47efca17a7bdd186cdd86897ddfa","Type":"ContainerStarted","Data":"3ae6e2eec063fb1f323edfe1732b1dec98223d1bdefb0b1eb1f80ca2bb772a87"} Jul 22 18:56:38.555444 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:38.555428 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 18:56:38.556670 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:38.556657 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Jul 22 18:56:38.569555 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:38.569520 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 18:56:38.597661 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:38.597639 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 18:56:39.272771 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.272739 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 18:56:39.421143 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.421103 2583 apiserver.go:52] "Watching apiserver" Jul 22 18:56:39.429187 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.429159 2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 18:56:39.431025 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.431002 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp","openshift-cluster-node-tuning-operator/tuned-9nsdt","openshift-multus/multus-additional-cni-plugins-t6qvj","openshift-network-diagnostics/network-check-target-bdzlg","openshift-ovn-kubernetes/ovnkube-node-66v5g","kube-system/global-pull-secret-syncer-j79dm","kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal","openshift-dns/node-resolver-2fd99","openshift-image-registry/node-ca-fxb82","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal","openshift-multus/multus-qtb5p","openshift-multus/network-metrics-daemon-p27ms","openshift-network-operator/iptables-alerter-7vpps","kube-system/konnectivity-agent-v2tfs"] Jul 22 18:56:39.432627 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.432601 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7vpps" Jul 22 18:56:39.434150 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.434108 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.435331 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.435286 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.435615 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.435507 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8whx7\"" Jul 22 18:56:39.435813 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.435791 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 18:56:39.435915 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.435888 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 18:56:39.436183 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.436165 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 18:56:39.436622 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.436607 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:39.436715 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:39.436695 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:56:39.437842 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.437806 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8fc79\"" Jul 22 18:56:39.437933 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.437893 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.437998 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.437938 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 18:56:39.438192 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.438173 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 18:56:39.438458 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.438436 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 18:56:39.438563 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.438510 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 18:56:39.438793 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.438772 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 18:56:39.438902 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.438861 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 18:56:39.438991 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.438974 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 18:56:39.439840 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.439821 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lxdd9\"" Jul 22 18:56:39.440240 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.440220 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:39.441031 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:39.440305 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:56:39.441031 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.440586 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 18:56:39.441031 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.440664 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 18:56:39.441031 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.440900 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 18:56:39.442086 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.442063 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 18:56:39.442407 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.442389 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 18:56:39.442613 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.442597 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 18:56:39.446487 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.443789 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 18:56:39.446487 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.443812 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bv2fg\"" Jul 22 18:56:39.446487 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.444525 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.446487 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.446189 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2fd99" Jul 22 18:56:39.447201 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.447177 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 18:56:39.447415 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.447396 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 18:56:39.447522 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.447503 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mhszb\"" Jul 22 18:56:39.447668 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.447579 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 18:56:39.448346 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.448320 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fxb82" Jul 22 18:56:39.448563 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.448546 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5mc9d\"" Jul 22 18:56:39.448912 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.448884 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 18:56:39.449020 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.448992 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 18:56:39.449827 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.449808 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v2tfs" Jul 22 18:56:39.451958 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.451928 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.452630 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.452584 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 18:56:39.452630 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.452597 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 18:56:39.452772 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.452668 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qtrcg\"" Jul 22 18:56:39.452955 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.452939 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:39.453020 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:39.452999 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:56:39.454334 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454310 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85d1fd76-3e08-49cc-8b59-81d1a2c69867-cni-binary-copy\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.454438 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454350 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-run-systemd\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.454438 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454378 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-run-ovn-kubernetes\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.454438 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454403 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-cni-bin\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.454438 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454430 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-device-dir\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.454661 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454454 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-kubernetes\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.454661 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454479 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wx9g\" (UniqueName: \"kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g\") pod \"network-check-target-bdzlg\" (UID: \"e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc\") " pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:39.454661 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454503 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-run-netns\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.454661 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454546 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-sysconfig\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.454661 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454591 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-env-overrides\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.454661 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454617 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-sysctl-conf\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.454935 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454681 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-var-lib-kubelet\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.454935 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454718 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85d1fd76-3e08-49cc-8b59-81d1a2c69867-system-cni-dir\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.454935 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454756 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g6ld\" (UniqueName: \"kubernetes.io/projected/85d1fd76-3e08-49cc-8b59-81d1a2c69867-kube-api-access-4g6ld\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.454935 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454783 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-etc-openvswitch\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.454935 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454820 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-run-openvswitch\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.454935 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454847 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.454935 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454873 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.454935 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454900 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-registration-dir\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.454935 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454923 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-host\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.455315 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454947 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd25db36-fb00-470e-b14a-17c696de8379-tmp\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.455315 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454973 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnx28\" (UniqueName: \"kubernetes.io/projected/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-kube-api-access-nnx28\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.455315 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.454997 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-modprobe-d\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.455315 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455024 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-run-ovn\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.455315 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455111 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 18:56:39.455315 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455117 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-lib-modules\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.455315 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455142 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fd25db36-fb00-470e-b14a-17c696de8379-etc-tuned\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.455315 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455169 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4g66\" (UniqueName: \"kubernetes.io/projected/fd25db36-fb00-470e-b14a-17c696de8379-kube-api-access-q4g66\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.455315 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455192 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da93be3f-70eb-44f9-bf6f-e25db9937a11-host-slash\") pod \"iptables-alerter-7vpps\" (UID: \"da93be3f-70eb-44f9-bf6f-e25db9937a11\") " pod="openshift-network-operator/iptables-alerter-7vpps" Jul 22 18:56:39.455315 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455230 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrqwx\" (UniqueName: \"kubernetes.io/projected/da93be3f-70eb-44f9-bf6f-e25db9937a11-kube-api-access-xrqwx\") pod \"iptables-alerter-7vpps\" (UID: \"da93be3f-70eb-44f9-bf6f-e25db9937a11\") " pod="openshift-network-operator/iptables-alerter-7vpps" Jul 22 18:56:39.455315 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455259 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85d1fd76-3e08-49cc-8b59-81d1a2c69867-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.455787 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455338 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-var-lib-openvswitch\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.455787 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455375 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-sysctl-d\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.455787 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455398 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-sys\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.455787 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455426 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/da93be3f-70eb-44f9-bf6f-e25db9937a11-iptables-alerter-script\") pod \"iptables-alerter-7vpps\" (UID: \"da93be3f-70eb-44f9-bf6f-e25db9937a11\") " pod="openshift-network-operator/iptables-alerter-7vpps" Jul 22 18:56:39.455787 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455456 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85d1fd76-3e08-49cc-8b59-81d1a2c69867-cnibin\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.455787 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455480 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-ovnkube-script-lib\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.455787 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455505 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret\") pod \"global-pull-secret-syncer-j79dm\" (UID: \"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd\") " pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:39.455787 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455530 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-sys-fs\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.455787 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455597 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-systemd\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.455787 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455621 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-ovn-node-metrics-cert\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.455787 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455649 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krhfj\" (UniqueName: \"kubernetes.io/projected/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-kube-api-access-krhfj\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.455787 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455673 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-slash\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.455787 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455700 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-ovnkube-config\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.455787 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455723 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5ldsz\"" Jul 22 18:56:39.456330 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455936 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 18:56:39.456330 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.456108 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 18:56:39.456330 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.455725 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-run\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.456330 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.456214 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85d1fd76-3e08-49cc-8b59-81d1a2c69867-os-release\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.456330 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.456242 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-node-log\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.456330 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.456295 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-log-socket\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.456330 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.456318 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7bcq5\"" Jul 22 18:56:39.456330 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.456320 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.456659 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.456344 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-kubelet\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.456659 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.456384 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-systemd-units\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.456659 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.456408 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-kubelet-config\") pod \"global-pull-secret-syncer-j79dm\" (UID: \"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd\") " pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:39.456659 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.456109 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 18:56:39.456659 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.456436 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-socket-dir\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.456659 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.456499 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85d1fd76-3e08-49cc-8b59-81d1a2c69867-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.456659 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.456547 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-dbus\") pod \"global-pull-secret-syncer-j79dm\" (UID: \"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd\") " pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:39.456659 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.456575 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85d1fd76-3e08-49cc-8b59-81d1a2c69867-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.456659 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.456601 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-cni-netd\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.458569 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.458547 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 18:56:39.486308 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.486282 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fkpfw" Jul 22 18:56:39.492757 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.492718 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fkpfw" Jul 22 18:56:39.545217 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.545132 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 18:56:39.557054 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557020 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-sysctl-conf\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.557255 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557063 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-var-lib-kubelet\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.557255 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557088 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85d1fd76-3e08-49cc-8b59-81d1a2c69867-system-cni-dir\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.557255 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557115 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g6ld\" (UniqueName: \"kubernetes.io/projected/85d1fd76-3e08-49cc-8b59-81d1a2c69867-kube-api-access-4g6ld\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.557255 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557138 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-etc-openvswitch\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.557255 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557159 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-run-openvswitch\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.557255 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557169 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-var-lib-kubelet\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.557255 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557182 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.557255 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557219 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkrzg\" (UniqueName: \"kubernetes.io/projected/0961d651-b746-4c2e-85f5-d2a00c044e3a-kube-api-access-zkrzg\") pod \"node-resolver-2fd99\" (UID: \"0961d651-b746-4c2e-85f5-d2a00c044e3a\") " pod="openshift-dns/node-resolver-2fd99" Jul 22 18:56:39.557255 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557240 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-etc-openvswitch\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.557255 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557243 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557289 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-registration-dir\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557298 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557316 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-host\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557254 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-sysctl-conf\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557324 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-run-openvswitch\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557342 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd25db36-fb00-470e-b14a-17c696de8379-tmp\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557291 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557377 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85d1fd76-3e08-49cc-8b59-81d1a2c69867-system-cni-dir\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557389 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-registration-dir\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557418 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnx28\" (UniqueName: \"kubernetes.io/projected/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-kube-api-access-nnx28\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557422 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-host\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557454 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-modprobe-d\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557601 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-run-ovn\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557632 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gcrw\" (UniqueName: \"kubernetes.io/projected/bb72b4c7-6047-4aa2-9604-58e177e0e98d-kube-api-access-7gcrw\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557687 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-run-ovn\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557775 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-lib-modules\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.557803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557797 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-modprobe-d\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.558469 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557805 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fd25db36-fb00-470e-b14a-17c696de8379-etc-tuned\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.558469 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557904 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-lib-modules\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.558469 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557938 2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 18:56:39.558469 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.557945 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4g66\" (UniqueName: \"kubernetes.io/projected/fd25db36-fb00-470e-b14a-17c696de8379-kube-api-access-q4g66\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.558469 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558269 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da93be3f-70eb-44f9-bf6f-e25db9937a11-host-slash\") pod \"iptables-alerter-7vpps\" (UID: \"da93be3f-70eb-44f9-bf6f-e25db9937a11\") " pod="openshift-network-operator/iptables-alerter-7vpps" Jul 22 18:56:39.558469 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558302 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrqwx\" (UniqueName: \"kubernetes.io/projected/da93be3f-70eb-44f9-bf6f-e25db9937a11-kube-api-access-xrqwx\") pod \"iptables-alerter-7vpps\" (UID: \"da93be3f-70eb-44f9-bf6f-e25db9937a11\") " pod="openshift-network-operator/iptables-alerter-7vpps" Jul 22 18:56:39.558469 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558330 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85d1fd76-3e08-49cc-8b59-81d1a2c69867-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.558469 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558379 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da93be3f-70eb-44f9-bf6f-e25db9937a11-host-slash\") pod \"iptables-alerter-7vpps\" (UID: \"da93be3f-70eb-44f9-bf6f-e25db9937a11\") " pod="openshift-network-operator/iptables-alerter-7vpps" Jul 22 18:56:39.558469 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558412 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-var-lib-openvswitch\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.558469 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558465 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0961d651-b746-4c2e-85f5-d2a00c044e3a-tmp-dir\") pod \"node-resolver-2fd99\" (UID: \"0961d651-b746-4c2e-85f5-d2a00c044e3a\") " pod="openshift-dns/node-resolver-2fd99" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558489 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-sysctl-d\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558531 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-sys\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558555 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/da93be3f-70eb-44f9-bf6f-e25db9937a11-iptables-alerter-script\") pod \"iptables-alerter-7vpps\" (UID: \"da93be3f-70eb-44f9-bf6f-e25db9937a11\") " pod="openshift-network-operator/iptables-alerter-7vpps" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558556 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-var-lib-openvswitch\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558608 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-cnibin\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558636 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb72b4c7-6047-4aa2-9604-58e177e0e98d-cni-binary-copy\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558639 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-sys\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558676 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85d1fd76-3e08-49cc-8b59-81d1a2c69867-cnibin\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558686 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-sysctl-d\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558700 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-ovnkube-script-lib\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558719 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret\") pod \"global-pull-secret-syncer-j79dm\" (UID: \"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd\") " pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558737 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-sys-fs\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558751 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85d1fd76-3e08-49cc-8b59-81d1a2c69867-cnibin\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558761 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-systemd\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558788 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-ovn-node-metrics-cert\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558814 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fslff\" (UniqueName: \"kubernetes.io/projected/00c3d60d-32a0-41ce-9ce6-45b51a3dd7da-kube-api-access-fslff\") pod \"node-ca-fxb82\" (UID: \"00c3d60d-32a0-41ce-9ce6-45b51a3dd7da\") " pod="openshift-image-registry/node-ca-fxb82" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558841 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-system-cni-dir\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.559010 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:39.558851 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 18:56:39.560013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558898 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-multus-conf-dir\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.560013 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:39.558932 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret podName:5ac11ffa-90fb-40e6-a021-0984d4d8c9bd nodeName:}" failed. No retries permitted until 2026-07-22 18:56:40.058908301 +0000 UTC m=+2.971072364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret") pod "global-pull-secret-syncer-j79dm" (UID: "5ac11ffa-90fb-40e6-a021-0984d4d8c9bd") : object "kube-system"/"original-pull-secret" not registered Jul 22 18:56:39.560013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558955 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs\") pod \"network-metrics-daemon-p27ms\" (UID: \"abb50681-d581-483f-a16a-bccdb437e24e\") " pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:39.560013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558979 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-sys-fs\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.560013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.558988 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krhfj\" (UniqueName: \"kubernetes.io/projected/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-kube-api-access-krhfj\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.560013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559036 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-systemd\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.560013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559062 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-slash\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.560013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559011 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85d1fd76-3e08-49cc-8b59-81d1a2c69867-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.560013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559089 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-hostroot\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.560013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559174 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-etc-kubernetes\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.560013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559200 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-ovnkube-config\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.560013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559226 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-var-lib-cni-bin\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.560013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559251 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-run\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.560013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559274 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85d1fd76-3e08-49cc-8b59-81d1a2c69867-os-release\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.560013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559295 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-node-log\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.560013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559310 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/da93be3f-70eb-44f9-bf6f-e25db9937a11-iptables-alerter-script\") pod \"iptables-alerter-7vpps\" (UID: \"da93be3f-70eb-44f9-bf6f-e25db9937a11\") " pod="openshift-network-operator/iptables-alerter-7vpps" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559318 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-log-socket\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559353 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-log-socket\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559366 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-run-netns\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559395 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-ovnkube-script-lib\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559407 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559462 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85d1fd76-3e08-49cc-8b59-81d1a2c69867-os-release\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559493 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559514 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-node-log\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559541 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-slash\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559549 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-run\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559620 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-kubelet\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559642 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-kubelet\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559673 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-systemd-units\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559708 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-var-lib-kubelet\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559717 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-systemd-units\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559742 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb72b4c7-6047-4aa2-9604-58e177e0e98d-multus-daemon-config\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.560864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559773 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-kubelet-config\") pod \"global-pull-secret-syncer-j79dm\" (UID: \"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd\") " pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559799 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-socket-dir\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559826 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85d1fd76-3e08-49cc-8b59-81d1a2c69867-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559879 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-kubelet-config\") pod \"global-pull-secret-syncer-j79dm\" (UID: \"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd\") " pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559890 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-ovnkube-config\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559914 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00c3d60d-32a0-41ce-9ce6-45b51a3dd7da-host\") pod \"node-ca-fxb82\" (UID: \"00c3d60d-32a0-41ce-9ce6-45b51a3dd7da\") " pod="openshift-image-registry/node-ca-fxb82" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559941 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9c816fc-7154-4b8d-86e9-85ef121f423d-agent-certs\") pod \"konnectivity-agent-v2tfs\" (UID: \"f9c816fc-7154-4b8d-86e9-85ef121f423d\") " pod="kube-system/konnectivity-agent-v2tfs" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559970 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-multus-cni-dir\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.559993 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85d1fd76-3e08-49cc-8b59-81d1a2c69867-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560011 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-dbus\") pod \"global-pull-secret-syncer-j79dm\" (UID: \"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd\") " pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560038 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85d1fd76-3e08-49cc-8b59-81d1a2c69867-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560065 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-cni-netd\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560089 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9c816fc-7154-4b8d-86e9-85ef121f423d-konnectivity-ca\") pod \"konnectivity-agent-v2tfs\" (UID: \"f9c816fc-7154-4b8d-86e9-85ef121f423d\") " pod="kube-system/konnectivity-agent-v2tfs" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560088 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-socket-dir\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560114 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-dbus\") pod \"global-pull-secret-syncer-j79dm\" (UID: \"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd\") " pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560144 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-cni-netd\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560199 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-os-release\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.562179 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560228 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-run-multus-certs\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560255 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9jld\" (UniqueName: \"kubernetes.io/projected/abb50681-d581-483f-a16a-bccdb437e24e-kube-api-access-f9jld\") pod \"network-metrics-daemon-p27ms\" (UID: \"abb50681-d581-483f-a16a-bccdb437e24e\") " pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560285 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85d1fd76-3e08-49cc-8b59-81d1a2c69867-cni-binary-copy\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560312 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-run-systemd\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560339 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-run-ovn-kubernetes\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560364 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-cni-bin\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560391 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-run-k8s-cni-cncf-io\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560417 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-var-lib-cni-multus\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560449 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-device-dir\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560477 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-kubernetes\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560503 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wx9g\" (UniqueName: \"kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g\") pod \"network-check-target-bdzlg\" (UID: \"e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc\") " pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560548 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-run-netns\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560575 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-multus-socket-dir-parent\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560590 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85d1fd76-3e08-49cc-8b59-81d1a2c69867-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560601 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-sysconfig\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560657 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-env-overrides\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560720 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-kubernetes\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.562855 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.560970 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-run-netns\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.563483 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.561021 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00c3d60d-32a0-41ce-9ce6-45b51a3dd7da-serviceca\") pod \"node-ca-fxb82\" (UID: \"00c3d60d-32a0-41ce-9ce6-45b51a3dd7da\") " pod="openshift-image-registry/node-ca-fxb82" Jul 22 18:56:39.563483 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.561047 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0961d651-b746-4c2e-85f5-d2a00c044e3a-hosts-file\") pod \"node-resolver-2fd99\" (UID: \"0961d651-b746-4c2e-85f5-d2a00c044e3a\") " pod="openshift-dns/node-resolver-2fd99" Jul 22 18:56:39.563483 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.561146 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-run-systemd\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.563483 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.561171 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85d1fd76-3e08-49cc-8b59-81d1a2c69867-cni-binary-copy\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.563483 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.561183 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-run-ovn-kubernetes\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.563483 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.561243 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-host-cni-bin\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.563483 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.561288 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-device-dir\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.563483 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.561391 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fd25db36-fb00-470e-b14a-17c696de8379-etc-sysconfig\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.563483 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.561588 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd25db36-fb00-470e-b14a-17c696de8379-tmp\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.563483 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.561610 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-env-overrides\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.563483 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.561628 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fd25db36-fb00-470e-b14a-17c696de8379-etc-tuned\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.563483 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.561984 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-ovn-node-metrics-cert\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.576737 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.576671 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g6ld\" (UniqueName: \"kubernetes.io/projected/85d1fd76-3e08-49cc-8b59-81d1a2c69867-kube-api-access-4g6ld\") pod \"multus-additional-cni-plugins-t6qvj\" (UID: \"85d1fd76-3e08-49cc-8b59-81d1a2c69867\") " pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.576903 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:39.576868 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 18:56:39.576903 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:39.576894 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 18:56:39.577017 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:39.576914 2583 projected.go:196] Error preparing data for projected volume kube-api-access-6wx9g for pod openshift-network-diagnostics/network-check-target-bdzlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:56:39.577017 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:39.576988 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g podName:e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc nodeName:}" failed. No retries permitted until 2026-07-22 18:56:40.076966386 +0000 UTC m=+2.989130449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6wx9g" (UniqueName: "kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g") pod "network-check-target-bdzlg" (UID: "e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:56:39.577887 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.577856 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnx28\" (UniqueName: \"kubernetes.io/projected/58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e-kube-api-access-nnx28\") pod \"ovnkube-node-66v5g\" (UID: \"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.577887 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.577859 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrqwx\" (UniqueName: \"kubernetes.io/projected/da93be3f-70eb-44f9-bf6f-e25db9937a11-kube-api-access-xrqwx\") pod \"iptables-alerter-7vpps\" (UID: \"da93be3f-70eb-44f9-bf6f-e25db9937a11\") " pod="openshift-network-operator/iptables-alerter-7vpps" Jul 22 18:56:39.578148 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.578085 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4g66\" (UniqueName: \"kubernetes.io/projected/fd25db36-fb00-470e-b14a-17c696de8379-kube-api-access-q4g66\") pod \"tuned-9nsdt\" (UID: \"fd25db36-fb00-470e-b14a-17c696de8379\") " pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.580055 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.580031 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krhfj\" (UniqueName: \"kubernetes.io/projected/0b93c368-4fdf-4f79-8d6e-0a3a75f842d0-kube-api-access-krhfj\") pod \"aws-ebs-csi-driver-node-s7xvp\" (UID: \"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.662018 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.661971 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-run-netns\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662018 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662024 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-var-lib-kubelet\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662282 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662051 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb72b4c7-6047-4aa2-9604-58e177e0e98d-multus-daemon-config\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662282 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662077 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00c3d60d-32a0-41ce-9ce6-45b51a3dd7da-host\") pod \"node-ca-fxb82\" (UID: \"00c3d60d-32a0-41ce-9ce6-45b51a3dd7da\") " pod="openshift-image-registry/node-ca-fxb82" Jul 22 18:56:39.662282 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662100 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-run-netns\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662282 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662105 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9c816fc-7154-4b8d-86e9-85ef121f423d-agent-certs\") pod \"konnectivity-agent-v2tfs\" (UID: \"f9c816fc-7154-4b8d-86e9-85ef121f423d\") " pod="kube-system/konnectivity-agent-v2tfs" Jul 22 18:56:39.662282 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662165 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00c3d60d-32a0-41ce-9ce6-45b51a3dd7da-host\") pod \"node-ca-fxb82\" (UID: \"00c3d60d-32a0-41ce-9ce6-45b51a3dd7da\") " pod="openshift-image-registry/node-ca-fxb82" Jul 22 18:56:39.662282 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662106 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-var-lib-kubelet\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662282 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662187 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-multus-cni-dir\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662282 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662221 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9c816fc-7154-4b8d-86e9-85ef121f423d-konnectivity-ca\") pod \"konnectivity-agent-v2tfs\" (UID: \"f9c816fc-7154-4b8d-86e9-85ef121f423d\") " pod="kube-system/konnectivity-agent-v2tfs" Jul 22 18:56:39.662282 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662249 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-os-release\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662282 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662273 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-run-multus-certs\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662297 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9jld\" (UniqueName: \"kubernetes.io/projected/abb50681-d581-483f-a16a-bccdb437e24e-kube-api-access-f9jld\") pod \"network-metrics-daemon-p27ms\" (UID: \"abb50681-d581-483f-a16a-bccdb437e24e\") " pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:39.662821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662344 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-run-k8s-cni-cncf-io\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662406 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-var-lib-cni-multus\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662449 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-multus-socket-dir-parent\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662478 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00c3d60d-32a0-41ce-9ce6-45b51a3dd7da-serviceca\") pod \"node-ca-fxb82\" (UID: \"00c3d60d-32a0-41ce-9ce6-45b51a3dd7da\") " pod="openshift-image-registry/node-ca-fxb82" Jul 22 18:56:39.662821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662504 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0961d651-b746-4c2e-85f5-d2a00c044e3a-hosts-file\") pod \"node-resolver-2fd99\" (UID: \"0961d651-b746-4c2e-85f5-d2a00c044e3a\") " pod="openshift-dns/node-resolver-2fd99" Jul 22 18:56:39.662821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662558 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrzg\" (UniqueName: \"kubernetes.io/projected/0961d651-b746-4c2e-85f5-d2a00c044e3a-kube-api-access-zkrzg\") pod \"node-resolver-2fd99\" (UID: \"0961d651-b746-4c2e-85f5-d2a00c044e3a\") " pod="openshift-dns/node-resolver-2fd99" Jul 22 18:56:39.662821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662591 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gcrw\" (UniqueName: \"kubernetes.io/projected/bb72b4c7-6047-4aa2-9604-58e177e0e98d-kube-api-access-7gcrw\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662629 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0961d651-b746-4c2e-85f5-d2a00c044e3a-tmp-dir\") pod \"node-resolver-2fd99\" (UID: \"0961d651-b746-4c2e-85f5-d2a00c044e3a\") " pod="openshift-dns/node-resolver-2fd99" Jul 22 18:56:39.662821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662616 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-multus-cni-dir\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662666 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-cnibin\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662693 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb72b4c7-6047-4aa2-9604-58e177e0e98d-cni-binary-copy\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662721 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb72b4c7-6047-4aa2-9604-58e177e0e98d-multus-daemon-config\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662730 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fslff\" (UniqueName: \"kubernetes.io/projected/00c3d60d-32a0-41ce-9ce6-45b51a3dd7da-kube-api-access-fslff\") pod \"node-ca-fxb82\" (UID: \"00c3d60d-32a0-41ce-9ce6-45b51a3dd7da\") " pod="openshift-image-registry/node-ca-fxb82" Jul 22 18:56:39.662821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662785 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-system-cni-dir\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.662821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662812 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-multus-conf-dir\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662813 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-os-release\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662864 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs\") pod \"network-metrics-daemon-p27ms\" (UID: \"abb50681-d581-483f-a16a-bccdb437e24e\") " pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662895 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-hostroot\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662900 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0961d651-b746-4c2e-85f5-d2a00c044e3a-hosts-file\") pod \"node-resolver-2fd99\" (UID: \"0961d651-b746-4c2e-85f5-d2a00c044e3a\") " pod="openshift-dns/node-resolver-2fd99" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662917 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9c816fc-7154-4b8d-86e9-85ef121f423d-konnectivity-ca\") pod \"konnectivity-agent-v2tfs\" (UID: \"f9c816fc-7154-4b8d-86e9-85ef121f423d\") " pod="kube-system/konnectivity-agent-v2tfs" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662949 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-run-multus-certs\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.662944 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-etc-kubernetes\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.663024 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-cnibin\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.663081 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-var-lib-cni-multus\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.663094 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00c3d60d-32a0-41ce-9ce6-45b51a3dd7da-serviceca\") pod \"node-ca-fxb82\" (UID: \"00c3d60d-32a0-41ce-9ce6-45b51a3dd7da\") " pod="openshift-image-registry/node-ca-fxb82" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.663128 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-run-k8s-cni-cncf-io\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.663131 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-var-lib-cni-bin\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.663165 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-multus-socket-dir-parent\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.663213 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-hostroot\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.663242 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0961d651-b746-4c2e-85f5-d2a00c044e3a-tmp-dir\") pod \"node-resolver-2fd99\" (UID: \"0961d651-b746-4c2e-85f5-d2a00c044e3a\") " pod="openshift-dns/node-resolver-2fd99" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.663251 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-multus-conf-dir\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.663217 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-host-var-lib-cni-bin\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.663213 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-etc-kubernetes\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.663535 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:39.663279 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:56:39.664288 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.663317 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb72b4c7-6047-4aa2-9604-58e177e0e98d-system-cni-dir\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.664288 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:39.663346 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs podName:abb50681-d581-483f-a16a-bccdb437e24e nodeName:}" failed. No retries permitted until 2026-07-22 18:56:40.163325097 +0000 UTC m=+3.075489145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs") pod "network-metrics-daemon-p27ms" (UID: "abb50681-d581-483f-a16a-bccdb437e24e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:56:39.664288 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.663945 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb72b4c7-6047-4aa2-9604-58e177e0e98d-cni-binary-copy\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.665053 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.665033 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9c816fc-7154-4b8d-86e9-85ef121f423d-agent-certs\") pod \"konnectivity-agent-v2tfs\" (UID: \"f9c816fc-7154-4b8d-86e9-85ef121f423d\") " pod="kube-system/konnectivity-agent-v2tfs" Jul 22 18:56:39.677146 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.677115 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkrzg\" (UniqueName: \"kubernetes.io/projected/0961d651-b746-4c2e-85f5-d2a00c044e3a-kube-api-access-zkrzg\") pod \"node-resolver-2fd99\" (UID: \"0961d651-b746-4c2e-85f5-d2a00c044e3a\") " pod="openshift-dns/node-resolver-2fd99" Jul 22 18:56:39.677296 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.677219 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9jld\" (UniqueName: \"kubernetes.io/projected/abb50681-d581-483f-a16a-bccdb437e24e-kube-api-access-f9jld\") pod \"network-metrics-daemon-p27ms\" (UID: \"abb50681-d581-483f-a16a-bccdb437e24e\") " pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:39.677407 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.677387 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gcrw\" (UniqueName: \"kubernetes.io/projected/bb72b4c7-6047-4aa2-9604-58e177e0e98d-kube-api-access-7gcrw\") pod \"multus-qtb5p\" (UID: \"bb72b4c7-6047-4aa2-9604-58e177e0e98d\") " pod="openshift-multus/multus-qtb5p" Jul 22 18:56:39.677521 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.677492 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fslff\" (UniqueName: \"kubernetes.io/projected/00c3d60d-32a0-41ce-9ce6-45b51a3dd7da-kube-api-access-fslff\") pod \"node-ca-fxb82\" (UID: \"00c3d60d-32a0-41ce-9ce6-45b51a3dd7da\") " pod="openshift-image-registry/node-ca-fxb82" Jul 22 18:56:39.748335 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.748283 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7vpps" Jul 22 18:56:39.754881 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.754857 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" Jul 22 18:56:39.763897 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.763861 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t6qvj" Jul 22 18:56:39.769604 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.769583 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:56:39.776405 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.776375 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" Jul 22 18:56:39.784241 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.784221 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2fd99" Jul 22 18:56:39.791063 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.791044 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fxb82" Jul 22 18:56:39.797758 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.797716 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v2tfs" Jul 22 18:56:39.805485 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:39.805463 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qtb5p" Jul 22 18:56:40.065784 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:40.065694 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret\") pod \"global-pull-secret-syncer-j79dm\" (UID: \"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd\") " pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:40.065949 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:40.065832 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 18:56:40.065949 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:40.065902 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret podName:5ac11ffa-90fb-40e6-a021-0984d4d8c9bd nodeName:}" failed. No retries permitted until 2026-07-22 18:56:41.065882281 +0000 UTC m=+3.978046344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret") pod "global-pull-secret-syncer-j79dm" (UID: "5ac11ffa-90fb-40e6-a021-0984d4d8c9bd") : object "kube-system"/"original-pull-secret" not registered Jul 22 18:56:40.140137 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:40.140101 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58fc2f5c_a3b3_4b2b_a7d2_42d9394f5f6e.slice/crio-25a76f4d0978da9d5ffb885f5b174963808255c899430798d4c7367e23882e86 WatchSource:0}: Error finding container 25a76f4d0978da9d5ffb885f5b174963808255c899430798d4c7367e23882e86: Status 404 returned error can't find the container with id 25a76f4d0978da9d5ffb885f5b174963808255c899430798d4c7367e23882e86 Jul 22 18:56:40.152009 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:40.151900 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd25db36_fb00_470e_b14a_17c696de8379.slice/crio-017574f3886cf2547314195acf471dad036ffff494611dac232fd5795cf37e34 WatchSource:0}: Error finding container 017574f3886cf2547314195acf471dad036ffff494611dac232fd5795cf37e34: Status 404 returned error can't find the container with id 017574f3886cf2547314195acf471dad036ffff494611dac232fd5795cf37e34 Jul 22 18:56:40.153140 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:40.152990 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0961d651_b746_4c2e_85f5_d2a00c044e3a.slice/crio-01c12ac55955a2435e2b9b6f2911bea9888b95b60547c9510cc3b150423fe7e1 WatchSource:0}: Error finding container 01c12ac55955a2435e2b9b6f2911bea9888b95b60547c9510cc3b150423fe7e1: Status 404 returned error can't find the container with id 01c12ac55955a2435e2b9b6f2911bea9888b95b60547c9510cc3b150423fe7e1 Jul 22 18:56:40.154100 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:40.154072 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d1fd76_3e08_49cc_8b59_81d1a2c69867.slice/crio-7da02a2f02afc1249a0a5ff9fba1b626c6db6bb3775c20e0f8879ac9edb570d1 WatchSource:0}: Error finding container 7da02a2f02afc1249a0a5ff9fba1b626c6db6bb3775c20e0f8879ac9edb570d1: Status 404 returned error can't find the container with id 7da02a2f02afc1249a0a5ff9fba1b626c6db6bb3775c20e0f8879ac9edb570d1 Jul 22 18:56:40.155188 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:40.155160 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b93c368_4fdf_4f79_8d6e_0a3a75f842d0.slice/crio-841390036e528b3425e9aa99b4ca2893e53739c6e10f87a549bf153676b2f17a WatchSource:0}: Error finding container 841390036e528b3425e9aa99b4ca2893e53739c6e10f87a549bf153676b2f17a: Status 404 returned error can't find the container with id 841390036e528b3425e9aa99b4ca2893e53739c6e10f87a549bf153676b2f17a Jul 22 18:56:40.156311 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:56:40.156206 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c3d60d_32a0_41ce_9ce6_45b51a3dd7da.slice/crio-4b06d654aad6dbeec3709127751645e836cee058f25c3445499fd8b5f972d81d WatchSource:0}: Error finding container 4b06d654aad6dbeec3709127751645e836cee058f25c3445499fd8b5f972d81d: Status 404 returned error can't find the container with id 4b06d654aad6dbeec3709127751645e836cee058f25c3445499fd8b5f972d81d Jul 22 18:56:40.166888 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:40.166862 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wx9g\" (UniqueName: \"kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g\") pod \"network-check-target-bdzlg\" (UID: \"e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc\") " pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:40.166991 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:40.166934 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs\") pod \"network-metrics-daemon-p27ms\" (UID: \"abb50681-d581-483f-a16a-bccdb437e24e\") " pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:40.167055 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:40.167035 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 18:56:40.167169 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:40.167060 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 18:56:40.167169 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:40.167060 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:56:40.167169 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:40.167073 2583 projected.go:196] Error preparing data for projected volume kube-api-access-6wx9g for pod openshift-network-diagnostics/network-check-target-bdzlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:56:40.167169 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:40.167116 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs podName:abb50681-d581-483f-a16a-bccdb437e24e nodeName:}" failed. No retries permitted until 2026-07-22 18:56:41.167096762 +0000 UTC m=+4.079260832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs") pod "network-metrics-daemon-p27ms" (UID: "abb50681-d581-483f-a16a-bccdb437e24e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:56:40.167169 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:40.167137 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g podName:e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc nodeName:}" failed. No retries permitted until 2026-07-22 18:56:41.167127118 +0000 UTC m=+4.079291166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wx9g" (UniqueName: "kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g") pod "network-check-target-bdzlg" (UID: "e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:56:40.495442 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:40.495382 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 18:51:40 +0000 UTC" deadline="2028-04-17 10:00:48.986100128 +0000 UTC" Jul 22 18:56:40.495442 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:40.495438 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15231h4m8.490666796s" Jul 22 18:56:40.554842 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:40.554807 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fxb82" event={"ID":"00c3d60d-32a0-41ce-9ce6-45b51a3dd7da","Type":"ContainerStarted","Data":"4b06d654aad6dbeec3709127751645e836cee058f25c3445499fd8b5f972d81d"} Jul 22 18:56:40.556005 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:40.555976 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2fd99" event={"ID":"0961d651-b746-4c2e-85f5-d2a00c044e3a","Type":"ContainerStarted","Data":"01c12ac55955a2435e2b9b6f2911bea9888b95b60547c9510cc3b150423fe7e1"} Jul 22 18:56:40.557325 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:40.557270 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7vpps" event={"ID":"da93be3f-70eb-44f9-bf6f-e25db9937a11","Type":"ContainerStarted","Data":"54ef1b216015529d97c0f500ce781482159cb64bca793c20ff0d02475e2bd876"} Jul 22 18:56:40.558753 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:40.558725 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" event={"ID":"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0","Type":"ContainerStarted","Data":"841390036e528b3425e9aa99b4ca2893e53739c6e10f87a549bf153676b2f17a"} Jul 22 18:56:40.560764 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:40.560740 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" event={"ID":"fd25db36-fb00-470e-b14a-17c696de8379","Type":"ContainerStarted","Data":"017574f3886cf2547314195acf471dad036ffff494611dac232fd5795cf37e34"} Jul 22 18:56:40.562305 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:40.562278 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" event={"ID":"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e","Type":"ContainerStarted","Data":"25a76f4d0978da9d5ffb885f5b174963808255c899430798d4c7367e23882e86"} Jul 22 18:56:40.565121 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:40.564869 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" event={"ID":"f73e47efca17a7bdd186cdd86897ddfa","Type":"ContainerStarted","Data":"e9c2288f2de4dea8e62bb962d5885f61b5712d654fd46751ea858f8028bfbc3e"} Jul 22 18:56:40.566619 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:40.566593 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v2tfs" event={"ID":"f9c816fc-7154-4b8d-86e9-85ef121f423d","Type":"ContainerStarted","Data":"8e18a82f5407f7a3ea367d27cce5baa8d8e336fdfeafe2e82f8e9baa2f4fe43a"} Jul 22 18:56:40.569293 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:40.569264 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qtb5p" event={"ID":"bb72b4c7-6047-4aa2-9604-58e177e0e98d","Type":"ContainerStarted","Data":"4a002492de3ae02b77b7b014b5e87fca642b4927120e9c992c99b382f82b5f9c"} Jul 22 18:56:40.575227 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:40.575194 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6qvj" event={"ID":"85d1fd76-3e08-49cc-8b59-81d1a2c69867","Type":"ContainerStarted","Data":"7da02a2f02afc1249a0a5ff9fba1b626c6db6bb3775c20e0f8879ac9edb570d1"} Jul 22 18:56:40.582770 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:40.582700 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" podStartSLOduration=1.582557078 podStartE2EDuration="1.582557078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:56:40.582215838 +0000 UTC m=+3.494379911" watchObservedRunningTime="2026-07-22 18:56:40.582557078 +0000 UTC m=+3.494721151" Jul 22 18:56:41.074444 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:41.073498 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret\") pod \"global-pull-secret-syncer-j79dm\" (UID: \"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd\") " pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:41.074444 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:41.073835 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 18:56:41.074444 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:41.073914 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret podName:5ac11ffa-90fb-40e6-a021-0984d4d8c9bd nodeName:}" failed. No retries permitted until 2026-07-22 18:56:43.073892361 +0000 UTC m=+5.986056412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret") pod "global-pull-secret-syncer-j79dm" (UID: "5ac11ffa-90fb-40e6-a021-0984d4d8c9bd") : object "kube-system"/"original-pull-secret" not registered Jul 22 18:56:41.174832 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:41.174789 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wx9g\" (UniqueName: \"kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g\") pod \"network-check-target-bdzlg\" (UID: \"e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc\") " pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:41.175032 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:41.174894 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs\") pod \"network-metrics-daemon-p27ms\" (UID: \"abb50681-d581-483f-a16a-bccdb437e24e\") " pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:41.175100 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:41.175045 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:56:41.175155 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:41.175113 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs podName:abb50681-d581-483f-a16a-bccdb437e24e nodeName:}" failed. No retries permitted until 2026-07-22 18:56:43.175093132 +0000 UTC m=+6.087257201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs") pod "network-metrics-daemon-p27ms" (UID: "abb50681-d581-483f-a16a-bccdb437e24e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:56:41.175754 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:41.175584 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 18:56:41.175754 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:41.175612 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 18:56:41.175754 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:41.175625 2583 projected.go:196] Error preparing data for projected volume kube-api-access-6wx9g for pod openshift-network-diagnostics/network-check-target-bdzlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:56:41.175754 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:41.175705 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g podName:e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc nodeName:}" failed. No retries permitted until 2026-07-22 18:56:43.175686738 +0000 UTC m=+6.087850901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wx9g" (UniqueName: "kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g") pod "network-check-target-bdzlg" (UID: "e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:56:41.496362 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:41.496249 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 18:51:40 +0000 UTC" deadline="2028-03-31 21:07:05.253290246 +0000 UTC" Jul 22 18:56:41.496362 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:41.496297 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14834h10m23.756998227s" Jul 22 18:56:41.554160 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:41.546408 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:41.554160 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:41.546563 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:56:41.554160 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:41.546684 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:41.554160 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:41.546759 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:56:41.554160 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:41.546826 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:41.554160 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:41.546908 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:56:43.094972 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:43.094925 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret\") pod \"global-pull-secret-syncer-j79dm\" (UID: \"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd\") " pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:43.095369 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:43.095102 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 18:56:43.095369 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:43.095166 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret podName:5ac11ffa-90fb-40e6-a021-0984d4d8c9bd nodeName:}" failed. No retries permitted until 2026-07-22 18:56:47.095146546 +0000 UTC m=+10.007310615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret") pod "global-pull-secret-syncer-j79dm" (UID: "5ac11ffa-90fb-40e6-a021-0984d4d8c9bd") : object "kube-system"/"original-pull-secret" not registered Jul 22 18:56:43.196308 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:43.196260 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wx9g\" (UniqueName: \"kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g\") pod \"network-check-target-bdzlg\" (UID: \"e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc\") " pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:43.196516 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:43.196353 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs\") pod \"network-metrics-daemon-p27ms\" (UID: \"abb50681-d581-483f-a16a-bccdb437e24e\") " pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:43.196516 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:43.196487 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:56:43.196671 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:43.196576 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs podName:abb50681-d581-483f-a16a-bccdb437e24e nodeName:}" failed. No retries permitted until 2026-07-22 18:56:47.196551982 +0000 UTC m=+10.108716058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs") pod "network-metrics-daemon-p27ms" (UID: "abb50681-d581-483f-a16a-bccdb437e24e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:56:43.197045 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:43.197023 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 18:56:43.197156 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:43.197050 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 18:56:43.197156 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:43.197066 2583 projected.go:196] Error preparing data for projected volume kube-api-access-6wx9g for pod openshift-network-diagnostics/network-check-target-bdzlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:56:43.197156 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:43.197115 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g podName:e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc nodeName:}" failed. No retries permitted until 2026-07-22 18:56:47.197100148 +0000 UTC m=+10.109264218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wx9g" (UniqueName: "kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g") pod "network-check-target-bdzlg" (UID: "e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:56:43.545732 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:43.545697 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:43.546035 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:43.545842 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:56:43.546282 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:43.546260 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:43.546467 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:43.546372 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:56:43.546595 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:43.546576 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:43.547719 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:43.546681 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:56:45.544302 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:45.544048 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:45.544769 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:45.544388 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:56:45.544769 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:45.544172 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:45.544769 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:45.544463 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:56:45.544769 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:45.544052 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:45.544769 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:45.544533 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:56:47.135878 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:47.135827 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret\") pod \"global-pull-secret-syncer-j79dm\" (UID: \"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd\") " pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:47.136384 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:47.136009 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 18:56:47.136384 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:47.136088 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret podName:5ac11ffa-90fb-40e6-a021-0984d4d8c9bd nodeName:}" failed. No retries permitted until 2026-07-22 18:56:55.136070767 +0000 UTC m=+18.048234818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret") pod "global-pull-secret-syncer-j79dm" (UID: "5ac11ffa-90fb-40e6-a021-0984d4d8c9bd") : object "kube-system"/"original-pull-secret" not registered Jul 22 18:56:47.237300 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:47.237215 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wx9g\" (UniqueName: \"kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g\") pod \"network-check-target-bdzlg\" (UID: \"e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc\") " pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:47.237300 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:47.237312 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs\") pod \"network-metrics-daemon-p27ms\" (UID: \"abb50681-d581-483f-a16a-bccdb437e24e\") " pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:47.237620 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:47.237368 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 18:56:47.237620 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:47.237395 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 18:56:47.237620 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:47.237404 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:56:47.237620 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:47.237409 2583 projected.go:196] Error preparing data for projected volume kube-api-access-6wx9g for pod openshift-network-diagnostics/network-check-target-bdzlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:56:47.237620 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:47.237461 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g podName:e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc nodeName:}" failed. No retries permitted until 2026-07-22 18:56:55.237444798 +0000 UTC m=+18.149608846 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wx9g" (UniqueName: "kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g") pod "network-check-target-bdzlg" (UID: "e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:56:47.237620 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:47.237485 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs podName:abb50681-d581-483f-a16a-bccdb437e24e nodeName:}" failed. No retries permitted until 2026-07-22 18:56:55.237473067 +0000 UTC m=+18.149637121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs") pod "network-metrics-daemon-p27ms" (UID: "abb50681-d581-483f-a16a-bccdb437e24e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:56:47.544839 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:47.544799 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:47.544839 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:47.544837 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:47.545111 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:47.544800 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:47.545111 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:47.544946 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:56:47.545111 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:47.545023 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:56:47.545299 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:47.545113 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:56:49.544378 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:49.543939 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:49.544378 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:49.543981 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:49.544378 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:49.544071 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:56:49.544378 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:49.544117 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:49.544378 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:49.544191 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:56:49.544378 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:49.544280 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:56:51.544173 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:51.544134 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:51.544606 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:51.544282 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:56:51.544606 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:51.544134 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:51.544606 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:51.544376 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:56:51.544606 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:51.544134 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:51.544606 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:51.544447 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:56:53.544532 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:53.544480 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:53.544532 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:53.544532 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:53.545022 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:53.544480 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:53.545022 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:53.544625 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:56:53.545022 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:53.544722 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:56:53.545022 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:53.544799 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:56:55.202921 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:55.202880 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret\") pod \"global-pull-secret-syncer-j79dm\" (UID: \"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd\") " pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:55.203370 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:55.203009 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 18:56:55.203370 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:55.203061 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret podName:5ac11ffa-90fb-40e6-a021-0984d4d8c9bd nodeName:}" failed. No retries permitted until 2026-07-22 18:57:11.203047554 +0000 UTC m=+34.115211603 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret") pod "global-pull-secret-syncer-j79dm" (UID: "5ac11ffa-90fb-40e6-a021-0984d4d8c9bd") : object "kube-system"/"original-pull-secret" not registered Jul 22 18:56:55.304285 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:55.304233 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs\") pod \"network-metrics-daemon-p27ms\" (UID: \"abb50681-d581-483f-a16a-bccdb437e24e\") " pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:55.304444 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:55.304309 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wx9g\" (UniqueName: \"kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g\") pod \"network-check-target-bdzlg\" (UID: \"e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc\") " pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:55.304444 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:55.304404 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:56:55.304553 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:55.304447 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 18:56:55.304553 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:55.304465 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 18:56:55.304553 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:55.304472 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs podName:abb50681-d581-483f-a16a-bccdb437e24e nodeName:}" failed. No retries permitted until 2026-07-22 18:57:11.304456388 +0000 UTC m=+34.216620434 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs") pod "network-metrics-daemon-p27ms" (UID: "abb50681-d581-483f-a16a-bccdb437e24e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:56:55.304553 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:55.304477 2583 projected.go:196] Error preparing data for projected volume kube-api-access-6wx9g for pod openshift-network-diagnostics/network-check-target-bdzlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:56:55.304553 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:55.304546 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g podName:e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc nodeName:}" failed. No retries permitted until 2026-07-22 18:57:11.304530738 +0000 UTC m=+34.216694800 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wx9g" (UniqueName: "kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g") pod "network-check-target-bdzlg" (UID: "e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:56:55.544331 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:55.544294 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:55.544511 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:55.544340 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:55.544511 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:55.544294 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:55.544511 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:55.544430 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:56:55.544669 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:55.544516 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:56:55.544669 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:55.544616 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:56:57.544403 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:57.544363 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:57.544859 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:57.544466 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:56:57.544859 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:57.544539 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:57.544859 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:57.544632 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:56:57.544859 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:57.544646 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:57.544859 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:57.544755 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:56:59.544812 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:59.544776 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:56:59.545475 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:59.544776 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:56:59.545475 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:59.544916 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:56:59.545475 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:59.544968 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:56:59.545475 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:56:59.544776 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:56:59.545475 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:56:59.545083 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:57:00.652360 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:00.652216 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" event={"ID":"fd25db36-fb00-470e-b14a-17c696de8379","Type":"ContainerStarted","Data":"51136f6a285d4f632fd23eed2b25fa4c07e227dea31c6c3518db492f4ce88944"} Jul 22 18:57:00.657307 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:00.657280 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" event={"ID":"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e","Type":"ContainerStarted","Data":"9c5b9c5e4ae5b3291d76e923532004d4ad80149aaa1efbb86c2e93230e894147"} Jul 22 18:57:00.660192 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:00.660162 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qtb5p" event={"ID":"bb72b4c7-6047-4aa2-9604-58e177e0e98d","Type":"ContainerStarted","Data":"8a06c6f5d8c543c9b6cc5db4c84b1d2842db96d2d7c1b3e602551cb33a479985"} Jul 22 18:57:00.670133 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:00.669937 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9nsdt" podStartSLOduration=2.399187302 podStartE2EDuration="22.669917386s" totalImagesPullingTime="20.270730084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:00.669421912 +0000 UTC m=+23.581585981" watchObservedRunningTime="2026-07-22 18:57:00.669917386 +0000 UTC m=+23.582081455" Jul 22 18:57:00.690814 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:00.690770 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qtb5p" podStartSLOduration=2.39319476 podStartE2EDuration="22.690753518s" totalImagesPullingTime="20.297558758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:00.690736381 +0000 UTC m=+23.602900454" watchObservedRunningTime="2026-07-22 18:57:00.690753518 +0000 UTC m=+23.602917585" Jul 22 18:57:01.543971 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.543942 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:57:01.544126 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:01.544072 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:57:01.544126 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.543942 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:57:01.544221 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.543942 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:57:01.544221 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:01.544198 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:57:01.544304 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:01.544276 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:57:01.663681 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.663646 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7vpps" event={"ID":"da93be3f-70eb-44f9-bf6f-e25db9937a11","Type":"ContainerStarted","Data":"0b54e0224df6489ad5ce382eedaa506f2fa698fbc4702845a05e3faa9935b66e"} Jul 22 18:57:01.665154 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.665101 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" event={"ID":"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0","Type":"ContainerStarted","Data":"2a6335c3b6b99bbb4a08fa635a5aae643cf130632af03cbe6b3e606cea04da9e"} Jul 22 18:57:01.670152 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.670127 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" event={"ID":"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e","Type":"ContainerStarted","Data":"400cef4440f303832e32866b899202d4d6d80d463efb76e52e7d5eaa0cece008"} Jul 22 18:57:01.670287 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.670211 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" event={"ID":"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e","Type":"ContainerStarted","Data":"321c63f3003468d646e60da8e16139474ae4c2bd5dea96bd33a3af4658470cf6"} Jul 22 18:57:01.670287 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.670266 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" event={"ID":"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e","Type":"ContainerStarted","Data":"16541353a749b78a7044514364684d51ffc877380f8e5fe00334923f31cf4fef"} Jul 22 18:57:01.670389 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.670314 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" event={"ID":"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e","Type":"ContainerStarted","Data":"fcb123d33e9a9053d62facda54c1ed83f81cf69697507eede81682c4958d3fd0"} Jul 22 18:57:01.670389 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.670347 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" event={"ID":"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e","Type":"ContainerStarted","Data":"e66b8c399feb310c901c5f2ec6c7c08b8906da355bf8ffef1d0705f8af0013b9"} Jul 22 18:57:01.671472 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.671450 2583 generic.go:356] "Generic (PLEG): container finished" podID="9d89308905f53d9806708f28d8cb1d8c" containerID="b95fc7cf987a92df3c1be77cabe57f26e32662622fa6045610680c12d2d0970a" exitCode=0 Jul 22 18:57:01.671633 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.671533 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" event={"ID":"9d89308905f53d9806708f28d8cb1d8c","Type":"ContainerDied","Data":"b95fc7cf987a92df3c1be77cabe57f26e32662622fa6045610680c12d2d0970a"} Jul 22 18:57:01.672920 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.672901 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v2tfs" event={"ID":"f9c816fc-7154-4b8d-86e9-85ef121f423d","Type":"ContainerStarted","Data":"323c4d391ec5b30d8f32852ba252fe17090dc51caf4875798f6720c6f8e51c4f"} Jul 22 18:57:01.674182 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.674161 2583 generic.go:356] "Generic (PLEG): container finished" podID="85d1fd76-3e08-49cc-8b59-81d1a2c69867" containerID="164ebdc8471f1fb4344b25af0c62e571189627f4018139b90aafe6bbfb45eeed" exitCode=0 Jul 22 18:57:01.674264 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.674222 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6qvj" event={"ID":"85d1fd76-3e08-49cc-8b59-81d1a2c69867","Type":"ContainerDied","Data":"164ebdc8471f1fb4344b25af0c62e571189627f4018139b90aafe6bbfb45eeed"} Jul 22 18:57:01.675617 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.675594 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fxb82" event={"ID":"00c3d60d-32a0-41ce-9ce6-45b51a3dd7da","Type":"ContainerStarted","Data":"ab5bb919a5550ac314c4766ba877ed2f76d446cbf497b04629867365255b9e80"} Jul 22 18:57:01.676954 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.676929 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2fd99" event={"ID":"0961d651-b746-4c2e-85f5-d2a00c044e3a","Type":"ContainerStarted","Data":"d6f417384220684d25d0cc59d104b267289e1cadd6209100897294456b9e3d7c"} Jul 22 18:57:01.678015 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.677975 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7vpps" podStartSLOduration=3.4496399589999998 podStartE2EDuration="23.677964094s" totalImagesPullingTime="20.228324135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:01.677789713 +0000 UTC m=+24.589953780" watchObservedRunningTime="2026-07-22 18:57:01.677964094 +0000 UTC m=+24.590128162" Jul 22 18:57:01.711609 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.711401 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2fd99" podStartSLOduration=7.170746643 podStartE2EDuration="23.711385782s" totalImagesPullingTime="16.540639139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:01.710992055 +0000 UTC m=+24.623156122" watchObservedRunningTime="2026-07-22 18:57:01.711385782 +0000 UTC m=+24.623549849" Jul 22 18:57:01.728996 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.728942 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fxb82" podStartSLOduration=3.4963345 podStartE2EDuration="23.728927741s" totalImagesPullingTime="20.232593241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:01.728677777 +0000 UTC m=+24.640841845" watchObservedRunningTime="2026-07-22 18:57:01.728927741 +0000 UTC m=+24.641091810" Jul 22 18:57:01.782213 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.782161 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v2tfs" podStartSLOduration=7.248735802 podStartE2EDuration="23.78214693s" totalImagesPullingTime="16.533411128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:01.745880126 +0000 UTC m=+24.658044187" watchObservedRunningTime="2026-07-22 18:57:01.78214693 +0000 UTC m=+24.694311000" Jul 22 18:57:01.933773 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:01.933744 2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 18:57:02.515937 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:02.515829 2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T18:57:01.933768332Z","UUID":"ced83e5f-e81d-49d0-99d8-efe7e5e2ec93","Handler":null,"Name":"","Endpoint":""} Jul 22 18:57:02.517839 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:02.517813 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 18:57:02.517958 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:02.517861 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 18:57:02.682302 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:02.682256 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" event={"ID":"9d89308905f53d9806708f28d8cb1d8c","Type":"ContainerStarted","Data":"a05a22862b5040da099a0f80616ddb574abb060f9ef5f9eb4607386ea4f3109b"} Jul 22 18:57:02.685383 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:02.684987 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" event={"ID":"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0","Type":"ContainerStarted","Data":"27b9c58d80591fedb756da2ad1a7bbb4eb8986f457c0edfd883e11f2d8a553b6"} Jul 22 18:57:02.698904 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:02.698854 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" podStartSLOduration=23.698836817 podStartE2EDuration="23.698836817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:02.698721897 +0000 UTC m=+25.610885971" watchObservedRunningTime="2026-07-22 18:57:02.698836817 +0000 UTC m=+25.611000886" Jul 22 18:57:03.544951 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:03.544917 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:57:03.545244 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:03.544916 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:57:03.545244 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:03.545045 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:57:03.545244 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:03.545062 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:57:03.545244 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:03.545174 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:57:03.545410 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:03.545277 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:57:03.690028 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:03.689985 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" event={"ID":"0b93c368-4fdf-4f79-8d6e-0a3a75f842d0","Type":"ContainerStarted","Data":"dd13814da8e2e4fbb4a4a9e124372fdf0d78ecf2b7ebd8a02414cd50e2ba5cd0"} Jul 22 18:57:03.693840 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:03.693807 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" event={"ID":"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e","Type":"ContainerStarted","Data":"7e987c6886306c6bb027a76fb48f08574346e05fe3e421016dce13d4c2bfa97f"} Jul 22 18:57:03.708326 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:03.708278 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xvp" podStartSLOduration=3.398811082 podStartE2EDuration="25.70825897s" totalImagesPullingTime="22.309447888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:56:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:03.708103504 +0000 UTC m=+26.620267572" watchObservedRunningTime="2026-07-22 18:57:03.70825897 +0000 UTC m=+26.620423040" Jul 22 18:57:04.154045 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:04.154002 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v2tfs" Jul 22 18:57:04.154570 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:04.154550 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v2tfs" Jul 22 18:57:04.696836 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:04.696787 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v2tfs" Jul 22 18:57:04.697285 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:04.697186 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v2tfs" Jul 22 18:57:05.544852 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:05.544819 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:57:05.545070 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:05.544831 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:57:05.545070 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:05.544955 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:57:05.545070 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:05.544839 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:57:05.545070 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:05.545004 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:57:05.545257 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:05.545103 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:57:06.703858 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:06.703558 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" event={"ID":"58fc2f5c-a3b3-4b2b-a7d2-42d9394f5f6e","Type":"ContainerStarted","Data":"0b109dd2f954da33c4fb38d60bf0e92c3d592b7faa23ceab7f5eba9371ac6992"} Jul 22 18:57:06.704457 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:06.703911 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:57:06.704457 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:06.703977 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:57:06.704457 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:06.704015 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:57:06.706194 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:06.706169 2583 generic.go:356] "Generic (PLEG): container finished" podID="85d1fd76-3e08-49cc-8b59-81d1a2c69867" containerID="e06e573cd42cb54f62595d4366f6dd88d9644290227f60b167918dd1896c0c37" exitCode=0 Jul 22 18:57:06.706303 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:06.706253 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6qvj" event={"ID":"85d1fd76-3e08-49cc-8b59-81d1a2c69867","Type":"ContainerDied","Data":"e06e573cd42cb54f62595d4366f6dd88d9644290227f60b167918dd1896c0c37"} Jul 22 18:57:06.720329 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:06.720309 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:57:06.720429 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:06.720421 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:57:06.730666 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:06.730628 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" podStartSLOduration=8.434310205 podStartE2EDuration="28.73061588s" totalImagesPullingTime="20.296305675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:06.730316565 +0000 UTC m=+29.642480633" watchObservedRunningTime="2026-07-22 18:57:06.73061588 +0000 UTC m=+29.642779949" Jul 22 18:57:07.546595 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:07.546559 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:57:07.546595 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:07.546569 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:57:07.546595 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:07.546569 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:57:07.546883 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:07.546665 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:57:07.546883 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:07.546706 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:57:07.546883 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:07.546785 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:57:08.320478 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:08.320290 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j79dm"] Jul 22 18:57:08.320970 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:08.320569 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:57:08.320970 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:08.320654 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:57:08.323852 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:08.323819 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p27ms"] Jul 22 18:57:08.323992 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:08.323930 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:57:08.324052 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:08.324031 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:57:08.324607 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:08.324582 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bdzlg"] Jul 22 18:57:08.324741 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:08.324677 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:57:08.324804 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:08.324756 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:57:08.715131 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:08.715095 2583 generic.go:356] "Generic (PLEG): container finished" podID="85d1fd76-3e08-49cc-8b59-81d1a2c69867" containerID="5f889adb57b615acd7debd32d938fc7e24a7f56e8efb3eab1708502d2d5c49bd" exitCode=0 Jul 22 18:57:08.715309 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:08.715189 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6qvj" event={"ID":"85d1fd76-3e08-49cc-8b59-81d1a2c69867","Type":"ContainerDied","Data":"5f889adb57b615acd7debd32d938fc7e24a7f56e8efb3eab1708502d2d5c49bd"} Jul 22 18:57:09.547206 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:09.547180 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:57:09.547614 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:09.547283 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:57:09.547614 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:09.547342 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:57:09.547614 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:09.547449 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:57:10.543778 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:10.543752 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:57:10.543920 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:10.543852 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:57:10.722545 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:10.722514 2583 generic.go:356] "Generic (PLEG): container finished" podID="85d1fd76-3e08-49cc-8b59-81d1a2c69867" containerID="33bd0c24186de544a1c03ecf8d2093a511b9b2b00e8df4dd0d53383b32d7fe49" exitCode=0 Jul 22 18:57:10.722945 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:10.722572 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6qvj" event={"ID":"85d1fd76-3e08-49cc-8b59-81d1a2c69867","Type":"ContainerDied","Data":"33bd0c24186de544a1c03ecf8d2093a511b9b2b00e8df4dd0d53383b32d7fe49"} Jul 22 18:57:11.226326 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:11.226292 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret\") pod \"global-pull-secret-syncer-j79dm\" (UID: \"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd\") " pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:57:11.226482 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:11.226455 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 18:57:11.226565 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:11.226543 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret podName:5ac11ffa-90fb-40e6-a021-0984d4d8c9bd nodeName:}" failed. No retries permitted until 2026-07-22 18:57:43.226526095 +0000 UTC m=+66.138690140 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret") pod "global-pull-secret-syncer-j79dm" (UID: "5ac11ffa-90fb-40e6-a021-0984d4d8c9bd") : object "kube-system"/"original-pull-secret" not registered Jul 22 18:57:11.327722 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:11.327680 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wx9g\" (UniqueName: \"kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g\") pod \"network-check-target-bdzlg\" (UID: \"e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc\") " pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:57:11.327879 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:11.327760 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs\") pod \"network-metrics-daemon-p27ms\" (UID: \"abb50681-d581-483f-a16a-bccdb437e24e\") " pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:57:11.327879 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:11.327869 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:57:11.328001 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:11.327881 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 18:57:11.328001 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:11.327907 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 18:57:11.328001 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:11.327921 2583 projected.go:196] Error preparing data for projected volume kube-api-access-6wx9g for pod openshift-network-diagnostics/network-check-target-bdzlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:57:11.328001 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:11.327934 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs podName:abb50681-d581-483f-a16a-bccdb437e24e nodeName:}" failed. No retries permitted until 2026-07-22 18:57:43.327914188 +0000 UTC m=+66.240078253 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs") pod "network-metrics-daemon-p27ms" (UID: "abb50681-d581-483f-a16a-bccdb437e24e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 18:57:11.328001 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:11.327974 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g podName:e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc nodeName:}" failed. No retries permitted until 2026-07-22 18:57:43.32795566 +0000 UTC m=+66.240119725 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wx9g" (UniqueName: "kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g") pod "network-check-target-bdzlg" (UID: "e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 18:57:11.544022 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:11.543983 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:57:11.544183 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:11.544120 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdzlg" podUID="e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc" Jul 22 18:57:11.544229 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:11.544184 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:57:11.544311 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:11.544292 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p27ms" podUID="abb50681-d581-483f-a16a-bccdb437e24e" Jul 22 18:57:12.544478 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:12.544444 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:57:12.544887 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:12.544590 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j79dm" podUID="5ac11ffa-90fb-40e6-a021-0984d4d8c9bd" Jul 22 18:57:13.405986 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.405948 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeReady" Jul 22 18:57:13.406163 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.406096 2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 18:57:13.441302 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.441264 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c97f75d66-rdvdn"] Jul 22 18:57:13.446208 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.446183 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.449036 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.449009 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 18:57:13.449306 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.449280 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 18:57:13.449441 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.449327 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 18:57:13.449441 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.449407 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bx5gq\"" Jul 22 18:57:13.457059 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.457033 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 18:57:13.459704 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.458927 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-2fmdc"] Jul 22 18:57:13.463986 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.463963 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc"] Jul 22 18:57:13.464622 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.464235 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-2fmdc" Jul 22 18:57:13.468919 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.468044 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-scnm7"] Jul 22 18:57:13.468919 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.468171 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc" Jul 22 18:57:13.468919 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.468223 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nlpp9\"" Jul 22 18:57:13.468919 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.468457 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 22 18:57:13.470851 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.470828 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 22 18:57:13.471088 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.471070 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 22 18:57:13.471283 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.471267 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c97f75d66-rdvdn"] Jul 22 18:57:13.471373 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.471299 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn"] Jul 22 18:57:13.471438 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.471384 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:13.471512 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.471478 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 22 18:57:13.471695 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.471675 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5lxr2\"" Jul 22 18:57:13.471956 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.471937 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 22 18:57:13.472139 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.472124 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 22 18:57:13.476408 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.475661 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-2nd4q"] Jul 22 18:57:13.478428 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.476557 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn" Jul 22 18:57:13.478428 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.476746 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 22 18:57:13.478428 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.477240 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 22 18:57:13.482519 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.479422 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp"] Jul 22 18:57:13.482519 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.479611 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 22 18:57:13.482519 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.480104 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 22 18:57:13.482519 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.480351 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bltzr\"" Jul 22 18:57:13.482519 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.480804 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.482519 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.480825 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 22 18:57:13.482519 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.480804 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 22 18:57:13.482519 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.482104 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 22 18:57:13.482960 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.482794 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8494c9898d-4jsgr"] Jul 22 18:57:13.482960 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.482879 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" Jul 22 18:57:13.483892 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.483865 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mcxf4\"" Jul 22 18:57:13.484887 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.484865 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 22 18:57:13.485741 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.485721 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7"] Jul 22 18:57:13.485962 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.485925 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 22 18:57:13.486449 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.485985 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:13.490017 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.489017 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-2fmdc"] Jul 22 18:57:13.490017 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.489058 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-scnm7"] Jul 22 18:57:13.490017 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.489081 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc"] Jul 22 18:57:13.490017 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.489144 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zmgjp"] Jul 22 18:57:13.490993 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.490974 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 18:57:13.491266 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.491243 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 22 18:57:13.491266 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.490981 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 18:57:13.491604 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.491582 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zs4n4\"" Jul 22 18:57:13.491766 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.491750 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 22 18:57:13.491882 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.491865 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 22 18:57:13.492137 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.492120 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 22 18:57:13.492454 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.492436 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 18:57:13.492661 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.492150 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-w2dk4\"" Jul 22 18:57:13.492739 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.492665 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 22 18:57:13.492739 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.492678 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8l9qz\"" Jul 22 18:57:13.492739 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.491801 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 22 18:57:13.492891 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.492823 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 22 18:57:13.492891 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.491589 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 22 18:57:13.493032 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.493016 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 18:57:13.493120 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.493036 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:13.493334 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.493299 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 22 18:57:13.493621 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.493591 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dk7bn"] Jul 22 18:57:13.493823 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.493793 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zmgjp" Jul 22 18:57:13.495983 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.495958 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 22 18:57:13.496147 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.496127 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 18:57:13.496896 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.496880 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dm4b7\"" Jul 22 18:57:13.497255 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.497229 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mqrp5\"" Jul 22 18:57:13.497347 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.497277 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 18:57:13.497400 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.497380 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 22 18:57:13.497553 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.497530 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 18:57:13.497691 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.497674 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 18:57:13.497882 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.497674 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 18:57:13.497882 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.497866 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 22 18:57:13.498046 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.498011 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-qtmwq"] Jul 22 18:57:13.498673 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.498203 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:13.502569 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.502546 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 18:57:13.502677 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.502578 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 18:57:13.502677 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.502630 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cc98d\"" Jul 22 18:57:13.502849 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.502829 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn"] Jul 22 18:57:13.502923 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.502868 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-2nd4q"] Jul 22 18:57:13.502923 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.502894 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7"] Jul 22 18:57:13.502923 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.502907 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-qtmwq" Jul 22 18:57:13.502923 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.502914 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8494c9898d-4jsgr"] Jul 22 18:57:13.503101 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.502933 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zmgjp"] Jul 22 18:57:13.505288 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.505268 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp"] Jul 22 18:57:13.505966 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.505948 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-l4qwb\"" Jul 22 18:57:13.506481 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.506464 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 18:57:13.508253 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.508232 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 18:57:13.522695 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.522674 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-qtmwq"] Jul 22 18:57:13.523479 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.523458 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dk7bn"] Jul 22 18:57:13.544332 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.544299 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-bound-sa-token\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.544453 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.544358 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.544453 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.544387 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e07a3932-ff41-4d65-b28d-885e260eec9a-trusted-ca\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.544453 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.544411 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e07a3932-ff41-4d65-b28d-885e260eec9a-installation-pull-secrets\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.544453 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.544447 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhv9m\" (UniqueName: \"kubernetes.io/projected/ed584a0f-c300-43ea-9cb8-dc928e29f0e8-kube-api-access-rhv9m\") pod \"volume-data-source-validator-78cf4c6f6-2fmdc\" (UID: \"ed584a0f-c300-43ea-9cb8-dc928e29f0e8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-2fmdc" Jul 22 18:57:13.544851 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.544472 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-certificates\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.544851 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.544523 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f6gn\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-kube-api-access-6f6gn\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.544851 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.544581 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e07a3932-ff41-4d65-b28d-885e260eec9a-image-registry-private-configuration\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.544851 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.544635 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e07a3932-ff41-4d65-b28d-885e260eec9a-ca-trust-extracted\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.548867 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.548843 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:57:13.549202 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.549188 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:57:13.551619 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.551601 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 18:57:13.552412 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.552397 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g8nmg\"" Jul 22 18:57:13.552917 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.552894 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5knzw\"" Jul 22 18:57:13.645233 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645196 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d766722-198a-46e6-9175-5c13a9138778-serving-cert\") pod \"service-ca-operator-5bcbc94c78-5rgfc\" (UID: \"1d766722-198a-46e6-9175-5c13a9138778\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc" Jul 22 18:57:13.645233 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645238 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d5899683-5c18-4ee4-9878-2fec416d25a8-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-9rgk7\" (UID: \"d5899683-5c18-4ee4-9878-2fec416d25a8\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:13.645454 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645274 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hl9z\" (UniqueName: \"kubernetes.io/projected/d5899683-5c18-4ee4-9878-2fec416d25a8-kube-api-access-2hl9z\") pod \"cluster-monitoring-operator-76cf8966f8-9rgk7\" (UID: \"d5899683-5c18-4ee4-9878-2fec416d25a8\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:13.645454 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645304 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:13.645454 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645339 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f6gn\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-kube-api-access-6f6gn\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.645454 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645363 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0aa0a66-a4d6-4557-bf05-05c34c80c173-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.645454 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645384 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0aa0a66-a4d6-4557-bf05-05c34c80c173-service-ca-bundle\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.645454 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645410 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa91b8fa-def9-4745-b1ec-66df9c2adc5a-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-jhcsn\" (UID: \"aa91b8fa-def9-4745-b1ec-66df9c2adc5a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn" Jul 22 18:57:13.645454 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645434 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4n6p\" (UniqueName: \"kubernetes.io/projected/1d766722-198a-46e6-9175-5c13a9138778-kube-api-access-z4n6p\") pod \"service-ca-operator-5bcbc94c78-5rgfc\" (UID: \"1d766722-198a-46e6-9175-5c13a9138778\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc" Jul 22 18:57:13.645755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645513 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-default-certificate\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:13.645755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645553 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/255f4f29-f31e-4d05-8544-d16d69b75e34-serving-cert\") pod \"console-operator-565b7b569d-scnm7\" (UID: \"255f4f29-f31e-4d05-8544-d16d69b75e34\") " pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:13.645755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645576 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e0aa0a66-a4d6-4557-bf05-05c34c80c173-snapshots\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.645755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645593 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/255f4f29-f31e-4d05-8544-d16d69b75e34-config\") pod \"console-operator-565b7b569d-scnm7\" (UID: \"255f4f29-f31e-4d05-8544-d16d69b75e34\") " pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:13.645755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645610 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:13.645755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645630 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-stats-auth\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:13.645755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645654 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d7ca2a-6211-4cf0-af77-49663ce893c3-tmp-dir\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:13.645755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645716 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/255f4f29-f31e-4d05-8544-d16d69b75e34-trusted-ca\") pod \"console-operator-565b7b569d-scnm7\" (UID: \"255f4f29-f31e-4d05-8544-d16d69b75e34\") " pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:13.645755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645749 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzjzf\" (UniqueName: \"kubernetes.io/projected/2092fe4a-f842-4ebf-876e-3fe8b8ebb0a4-kube-api-access-kzjzf\") pod \"ingress-canary-zmgjp\" (UID: \"2092fe4a-f842-4ebf-876e-3fe8b8ebb0a4\") " pod="openshift-ingress-canary/ingress-canary-zmgjp" Jul 22 18:57:13.646095 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645768 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxxlm\" (UniqueName: \"kubernetes.io/projected/aa91b8fa-def9-4745-b1ec-66df9c2adc5a-kube-api-access-zxxlm\") pod \"kube-storage-version-migrator-operator-7658d9b749-jhcsn\" (UID: \"aa91b8fa-def9-4745-b1ec-66df9c2adc5a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn" Jul 22 18:57:13.646095 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645791 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e07a3932-ff41-4d65-b28d-885e260eec9a-ca-trust-extracted\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.646095 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645807 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1d766722-198a-46e6-9175-5c13a9138778-config\") pod \"service-ca-operator-5bcbc94c78-5rgfc\" (UID: \"1d766722-198a-46e6-9175-5c13a9138778\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc" Jul 22 18:57:13.646095 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645835 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9rgk7\" (UID: \"d5899683-5c18-4ee4-9878-2fec416d25a8\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:13.646095 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645933 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2092fe4a-f842-4ebf-876e-3fe8b8ebb0a4-cert\") pod \"ingress-canary-zmgjp\" (UID: \"2092fe4a-f842-4ebf-876e-3fe8b8ebb0a4\") " pod="openshift-ingress-canary/ingress-canary-zmgjp" Jul 22 18:57:13.646095 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645972 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-bound-sa-token\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.646095 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.645999 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpgx9\" (UniqueName: \"kubernetes.io/projected/e0aa0a66-a4d6-4557-bf05-05c34c80c173-kube-api-access-lpgx9\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.646095 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646029 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9d7ca2a-6211-4cf0-af77-49663ce893c3-config-volume\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:13.646095 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646064 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.646095 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646090 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e07a3932-ff41-4d65-b28d-885e260eec9a-installation-pull-secrets\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.646480 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646123 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e07a3932-ff41-4d65-b28d-885e260eec9a-ca-trust-extracted\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.646480 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646177 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhv9m\" (UniqueName: \"kubernetes.io/projected/ed584a0f-c300-43ea-9cb8-dc928e29f0e8-kube-api-access-rhv9m\") pod \"volume-data-source-validator-78cf4c6f6-2fmdc\" (UID: \"ed584a0f-c300-43ea-9cb8-dc928e29f0e8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-2fmdc" Jul 22 18:57:13.646480 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:13.646186 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 18:57:13.646480 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646197 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-certificates\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.646480 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646214 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz5zt\" (UniqueName: \"kubernetes.io/projected/d9d7ca2a-6211-4cf0-af77-49663ce893c3-kube-api-access-bz5zt\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:13.646480 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:13.646200 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c97f75d66-rdvdn: secret "image-registry-tls" not found Jul 22 18:57:13.646480 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646246 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0aa0a66-a4d6-4557-bf05-05c34c80c173-tmp\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.646480 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:13.646308 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls podName:e07a3932-ff41-4d65-b28d-885e260eec9a nodeName:}" failed. No retries permitted until 2026-07-22 18:57:14.146285618 +0000 UTC m=+37.058449683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls") pod "image-registry-c97f75d66-rdvdn" (UID: "e07a3932-ff41-4d65-b28d-885e260eec9a") : secret "image-registry-tls" not found Jul 22 18:57:13.646480 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646350 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59r7h\" (UniqueName: \"kubernetes.io/projected/dd794227-b4ac-45cb-ab35-ba8f431d0948-kube-api-access-59r7h\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:13.646480 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646378 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:13.646480 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646431 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e07a3932-ff41-4d65-b28d-885e260eec9a-image-registry-private-configuration\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.646480 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646461 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znfrv\" (UniqueName: \"kubernetes.io/projected/a479373f-89ba-4003-81f0-2eb33cb5c35a-kube-api-access-znfrv\") pod \"cluster-samples-operator-79b7c55f87-rq9jp\" (UID: \"a479373f-89ba-4003-81f0-2eb33cb5c35a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" Jul 22 18:57:13.646480 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646486 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa91b8fa-def9-4745-b1ec-66df9c2adc5a-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-jhcsn\" (UID: \"aa91b8fa-def9-4745-b1ec-66df9c2adc5a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn" Jul 22 18:57:13.647121 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646553 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwkf\" (UniqueName: \"kubernetes.io/projected/255f4f29-f31e-4d05-8544-d16d69b75e34-kube-api-access-hzwkf\") pod \"console-operator-565b7b569d-scnm7\" (UID: \"255f4f29-f31e-4d05-8544-d16d69b75e34\") " pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:13.647121 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646655 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frqzx\" (UniqueName: \"kubernetes.io/projected/3466f1e1-169b-4031-8f7a-a551062b79eb-kube-api-access-frqzx\") pod \"network-check-source-7bc5b6bf85-qtmwq\" (UID: \"3466f1e1-169b-4031-8f7a-a551062b79eb\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-qtmwq" Jul 22 18:57:13.647121 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646711 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e07a3932-ff41-4d65-b28d-885e260eec9a-trusted-ca\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.647121 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646739 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0aa0a66-a4d6-4557-bf05-05c34c80c173-serving-cert\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.647121 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646765 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rq9jp\" (UID: \"a479373f-89ba-4003-81f0-2eb33cb5c35a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" Jul 22 18:57:13.647121 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.646765 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-certificates\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.647761 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.647740 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e07a3932-ff41-4d65-b28d-885e260eec9a-trusted-ca\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.651263 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.651167 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e07a3932-ff41-4d65-b28d-885e260eec9a-installation-pull-secrets\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.651363 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.651173 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e07a3932-ff41-4d65-b28d-885e260eec9a-image-registry-private-configuration\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.653719 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.653701 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f6gn\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-kube-api-access-6f6gn\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.656487 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.656435 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-bound-sa-token\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:13.659531 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.659472 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhv9m\" (UniqueName: \"kubernetes.io/projected/ed584a0f-c300-43ea-9cb8-dc928e29f0e8-kube-api-access-rhv9m\") pod \"volume-data-source-validator-78cf4c6f6-2fmdc\" (UID: \"ed584a0f-c300-43ea-9cb8-dc928e29f0e8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-2fmdc" Jul 22 18:57:13.747402 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.747366 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9d7ca2a-6211-4cf0-af77-49663ce893c3-config-volume\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:13.747606 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.747435 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz5zt\" (UniqueName: \"kubernetes.io/projected/d9d7ca2a-6211-4cf0-af77-49663ce893c3-kube-api-access-bz5zt\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:13.747606 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.747461 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0aa0a66-a4d6-4557-bf05-05c34c80c173-tmp\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.747606 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.747482 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59r7h\" (UniqueName: \"kubernetes.io/projected/dd794227-b4ac-45cb-ab35-ba8f431d0948-kube-api-access-59r7h\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:13.747735 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.747622 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:13.747735 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.747690 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znfrv\" (UniqueName: \"kubernetes.io/projected/a479373f-89ba-4003-81f0-2eb33cb5c35a-kube-api-access-znfrv\") pod \"cluster-samples-operator-79b7c55f87-rq9jp\" (UID: \"a479373f-89ba-4003-81f0-2eb33cb5c35a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" Jul 22 18:57:13.747735 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.747721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa91b8fa-def9-4745-b1ec-66df9c2adc5a-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-jhcsn\" (UID: \"aa91b8fa-def9-4745-b1ec-66df9c2adc5a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn" Jul 22 18:57:13.747846 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.747768 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwkf\" (UniqueName: \"kubernetes.io/projected/255f4f29-f31e-4d05-8544-d16d69b75e34-kube-api-access-hzwkf\") pod \"console-operator-565b7b569d-scnm7\" (UID: \"255f4f29-f31e-4d05-8544-d16d69b75e34\") " pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:13.747846 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.747796 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqzx\" (UniqueName: \"kubernetes.io/projected/3466f1e1-169b-4031-8f7a-a551062b79eb-kube-api-access-frqzx\") pod \"network-check-source-7bc5b6bf85-qtmwq\" (UID: \"3466f1e1-169b-4031-8f7a-a551062b79eb\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-qtmwq" Jul 22 18:57:13.747846 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.747839 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0aa0a66-a4d6-4557-bf05-05c34c80c173-tmp\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.747954 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.747843 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0aa0a66-a4d6-4557-bf05-05c34c80c173-serving-cert\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.747954 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:13.747926 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 18:57:13.747954 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.747938 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rq9jp\" (UID: \"a479373f-89ba-4003-81f0-2eb33cb5c35a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" Jul 22 18:57:13.748051 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.747970 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d766722-198a-46e6-9175-5c13a9138778-serving-cert\") pod \"service-ca-operator-5bcbc94c78-5rgfc\" (UID: \"1d766722-198a-46e6-9175-5c13a9138778\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc" Jul 22 18:57:13.748051 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:13.747981 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls podName:d9d7ca2a-6211-4cf0-af77-49663ce893c3 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:14.247963238 +0000 UTC m=+37.160127289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls") pod "dns-default-dk7bn" (UID: "d9d7ca2a-6211-4cf0-af77-49663ce893c3") : secret "dns-default-metrics-tls" not found Jul 22 18:57:13.748134 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748120 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9d7ca2a-6211-4cf0-af77-49663ce893c3-config-volume\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:13.748366 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:13.748345 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 18:57:13.748460 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:13.748434 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls podName:a479373f-89ba-4003-81f0-2eb33cb5c35a nodeName:}" failed. No retries permitted until 2026-07-22 18:57:14.248417609 +0000 UTC m=+37.160581659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-rq9jp" (UID: "a479373f-89ba-4003-81f0-2eb33cb5c35a") : secret "samples-operator-tls" not found Jul 22 18:57:13.748554 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748460 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d5899683-5c18-4ee4-9878-2fec416d25a8-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-9rgk7\" (UID: \"d5899683-5c18-4ee4-9878-2fec416d25a8\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:13.748554 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748481 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hl9z\" (UniqueName: \"kubernetes.io/projected/d5899683-5c18-4ee4-9878-2fec416d25a8-kube-api-access-2hl9z\") pod \"cluster-monitoring-operator-76cf8966f8-9rgk7\" (UID: \"d5899683-5c18-4ee4-9878-2fec416d25a8\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:13.748554 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748521 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:13.748554 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748550 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0aa0a66-a4d6-4557-bf05-05c34c80c173-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.748761 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748576 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0aa0a66-a4d6-4557-bf05-05c34c80c173-service-ca-bundle\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.748761 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748603 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa91b8fa-def9-4745-b1ec-66df9c2adc5a-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-jhcsn\" (UID: \"aa91b8fa-def9-4745-b1ec-66df9c2adc5a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn" Jul 22 18:57:13.748761 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748629 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4n6p\" (UniqueName: \"kubernetes.io/projected/1d766722-198a-46e6-9175-5c13a9138778-kube-api-access-z4n6p\") pod \"service-ca-operator-5bcbc94c78-5rgfc\" (UID: \"1d766722-198a-46e6-9175-5c13a9138778\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc" Jul 22 18:57:13.748761 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748653 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-default-certificate\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:13.748761 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748688 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/255f4f29-f31e-4d05-8544-d16d69b75e34-serving-cert\") pod \"console-operator-565b7b569d-scnm7\" (UID: \"255f4f29-f31e-4d05-8544-d16d69b75e34\") " pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:13.748761 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748724 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e0aa0a66-a4d6-4557-bf05-05c34c80c173-snapshots\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.748761 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748751 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/255f4f29-f31e-4d05-8544-d16d69b75e34-config\") pod \"console-operator-565b7b569d-scnm7\" (UID: \"255f4f29-f31e-4d05-8544-d16d69b75e34\") " pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:13.749143 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748782 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:13.749143 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748808 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-stats-auth\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:13.749143 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748830 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d7ca2a-6211-4cf0-af77-49663ce893c3-tmp-dir\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:13.749143 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748856 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/255f4f29-f31e-4d05-8544-d16d69b75e34-trusted-ca\") pod \"console-operator-565b7b569d-scnm7\" (UID: \"255f4f29-f31e-4d05-8544-d16d69b75e34\") " pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:13.749143 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748884 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzjzf\" (UniqueName: \"kubernetes.io/projected/2092fe4a-f842-4ebf-876e-3fe8b8ebb0a4-kube-api-access-kzjzf\") pod \"ingress-canary-zmgjp\" (UID: \"2092fe4a-f842-4ebf-876e-3fe8b8ebb0a4\") " pod="openshift-ingress-canary/ingress-canary-zmgjp" Jul 22 18:57:13.749143 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748915 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxxlm\" (UniqueName: \"kubernetes.io/projected/aa91b8fa-def9-4745-b1ec-66df9c2adc5a-kube-api-access-zxxlm\") pod \"kube-storage-version-migrator-operator-7658d9b749-jhcsn\" (UID: \"aa91b8fa-def9-4745-b1ec-66df9c2adc5a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn" Jul 22 18:57:13.749143 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748948 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1d766722-198a-46e6-9175-5c13a9138778-config\") pod \"service-ca-operator-5bcbc94c78-5rgfc\" (UID: \"1d766722-198a-46e6-9175-5c13a9138778\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc" Jul 22 18:57:13.749143 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.748973 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9rgk7\" (UID: \"d5899683-5c18-4ee4-9878-2fec416d25a8\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:13.749143 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.749002 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2092fe4a-f842-4ebf-876e-3fe8b8ebb0a4-cert\") pod \"ingress-canary-zmgjp\" (UID: \"2092fe4a-f842-4ebf-876e-3fe8b8ebb0a4\") " pod="openshift-ingress-canary/ingress-canary-zmgjp" Jul 22 18:57:13.749143 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.749036 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpgx9\" (UniqueName: \"kubernetes.io/projected/e0aa0a66-a4d6-4557-bf05-05c34c80c173-kube-api-access-lpgx9\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.749633 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.749214 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa91b8fa-def9-4745-b1ec-66df9c2adc5a-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-jhcsn\" (UID: \"aa91b8fa-def9-4745-b1ec-66df9c2adc5a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn" Jul 22 18:57:13.749633 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:13.749290 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 18:57:13.749633 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.749291 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d5899683-5c18-4ee4-9878-2fec416d25a8-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-9rgk7\" (UID: \"d5899683-5c18-4ee4-9878-2fec416d25a8\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:13.749633 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:13.749337 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs podName:dd794227-b4ac-45cb-ab35-ba8f431d0948 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:14.249321594 +0000 UTC m=+37.161485657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs") pod "router-default-8494c9898d-4jsgr" (UID: "dd794227-b4ac-45cb-ab35-ba8f431d0948") : secret "router-metrics-certs-default" not found Jul 22 18:57:13.749929 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.749906 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0aa0a66-a4d6-4557-bf05-05c34c80c173-service-ca-bundle\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.750099 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.750060 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d7ca2a-6211-4cf0-af77-49663ce893c3-tmp-dir\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:13.750210 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:13.750189 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle podName:dd794227-b4ac-45cb-ab35-ba8f431d0948 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:14.250173249 +0000 UTC m=+37.162337310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle") pod "router-default-8494c9898d-4jsgr" (UID: "dd794227-b4ac-45cb-ab35-ba8f431d0948") : configmap references non-existent config key: service-ca.crt Jul 22 18:57:13.750210 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.750206 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0aa0a66-a4d6-4557-bf05-05c34c80c173-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.751139 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.750624 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/255f4f29-f31e-4d05-8544-d16d69b75e34-config\") pod \"console-operator-565b7b569d-scnm7\" (UID: \"255f4f29-f31e-4d05-8544-d16d69b75e34\") " pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:13.751139 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.750828 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa91b8fa-def9-4745-b1ec-66df9c2adc5a-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-jhcsn\" (UID: \"aa91b8fa-def9-4745-b1ec-66df9c2adc5a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn" Jul 22 18:57:13.751139 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:13.750928 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 18:57:13.751139 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:13.750979 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls podName:d5899683-5c18-4ee4-9878-2fec416d25a8 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:14.250965255 +0000 UTC m=+37.163129319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9rgk7" (UID: "d5899683-5c18-4ee4-9878-2fec416d25a8") : secret "cluster-monitoring-operator-tls" not found Jul 22 18:57:13.751424 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.751144 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/255f4f29-f31e-4d05-8544-d16d69b75e34-trusted-ca\") pod \"console-operator-565b7b569d-scnm7\" (UID: \"255f4f29-f31e-4d05-8544-d16d69b75e34\") " pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:13.751424 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.751158 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0aa0a66-a4d6-4557-bf05-05c34c80c173-serving-cert\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.751424 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.751342 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e0aa0a66-a4d6-4557-bf05-05c34c80c173-snapshots\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.751610 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.751426 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1d766722-198a-46e6-9175-5c13a9138778-config\") pod \"service-ca-operator-5bcbc94c78-5rgfc\" (UID: \"1d766722-198a-46e6-9175-5c13a9138778\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc" Jul 22 18:57:13.751666 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.751632 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d766722-198a-46e6-9175-5c13a9138778-serving-cert\") pod \"service-ca-operator-5bcbc94c78-5rgfc\" (UID: \"1d766722-198a-46e6-9175-5c13a9138778\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc" Jul 22 18:57:13.752913 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.752879 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-stats-auth\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:13.753412 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.753375 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2092fe4a-f842-4ebf-876e-3fe8b8ebb0a4-cert\") pod \"ingress-canary-zmgjp\" (UID: \"2092fe4a-f842-4ebf-876e-3fe8b8ebb0a4\") " pod="openshift-ingress-canary/ingress-canary-zmgjp" Jul 22 18:57:13.753604 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.753585 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/255f4f29-f31e-4d05-8544-d16d69b75e34-serving-cert\") pod \"console-operator-565b7b569d-scnm7\" (UID: \"255f4f29-f31e-4d05-8544-d16d69b75e34\") " pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:13.753690 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.753675 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-default-certificate\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:13.760702 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.758604 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz5zt\" (UniqueName: \"kubernetes.io/projected/d9d7ca2a-6211-4cf0-af77-49663ce893c3-kube-api-access-bz5zt\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:13.760702 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.760170 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4n6p\" (UniqueName: \"kubernetes.io/projected/1d766722-198a-46e6-9175-5c13a9138778-kube-api-access-z4n6p\") pod \"service-ca-operator-5bcbc94c78-5rgfc\" (UID: \"1d766722-198a-46e6-9175-5c13a9138778\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc" Jul 22 18:57:13.760702 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.760402 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znfrv\" (UniqueName: \"kubernetes.io/projected/a479373f-89ba-4003-81f0-2eb33cb5c35a-kube-api-access-znfrv\") pod \"cluster-samples-operator-79b7c55f87-rq9jp\" (UID: \"a479373f-89ba-4003-81f0-2eb33cb5c35a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" Jul 22 18:57:13.760702 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.760460 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwkf\" (UniqueName: \"kubernetes.io/projected/255f4f29-f31e-4d05-8544-d16d69b75e34-kube-api-access-hzwkf\") pod \"console-operator-565b7b569d-scnm7\" (UID: \"255f4f29-f31e-4d05-8544-d16d69b75e34\") " pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:13.761741 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.761174 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzjzf\" (UniqueName: \"kubernetes.io/projected/2092fe4a-f842-4ebf-876e-3fe8b8ebb0a4-kube-api-access-kzjzf\") pod \"ingress-canary-zmgjp\" (UID: \"2092fe4a-f842-4ebf-876e-3fe8b8ebb0a4\") " pod="openshift-ingress-canary/ingress-canary-zmgjp" Jul 22 18:57:13.761741 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.761700 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59r7h\" (UniqueName: \"kubernetes.io/projected/dd794227-b4ac-45cb-ab35-ba8f431d0948-kube-api-access-59r7h\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:13.762046 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.761942 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hl9z\" (UniqueName: \"kubernetes.io/projected/d5899683-5c18-4ee4-9878-2fec416d25a8-kube-api-access-2hl9z\") pod \"cluster-monitoring-operator-76cf8966f8-9rgk7\" (UID: \"d5899683-5c18-4ee4-9878-2fec416d25a8\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:13.762046 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.761948 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpgx9\" (UniqueName: \"kubernetes.io/projected/e0aa0a66-a4d6-4557-bf05-05c34c80c173-kube-api-access-lpgx9\") pod \"insights-operator-555d9d7885-2nd4q\" (UID: \"e0aa0a66-a4d6-4557-bf05-05c34c80c173\") " pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.762906 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.762887 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxxlm\" (UniqueName: \"kubernetes.io/projected/aa91b8fa-def9-4745-b1ec-66df9c2adc5a-kube-api-access-zxxlm\") pod \"kube-storage-version-migrator-operator-7658d9b749-jhcsn\" (UID: \"aa91b8fa-def9-4745-b1ec-66df9c2adc5a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn" Jul 22 18:57:13.763280 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.763263 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqzx\" (UniqueName: \"kubernetes.io/projected/3466f1e1-169b-4031-8f7a-a551062b79eb-kube-api-access-frqzx\") pod \"network-check-source-7bc5b6bf85-qtmwq\" (UID: \"3466f1e1-169b-4031-8f7a-a551062b79eb\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-qtmwq" Jul 22 18:57:13.777730 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.777708 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-2fmdc" Jul 22 18:57:13.795535 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.795514 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc" Jul 22 18:57:13.804292 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.804269 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:13.813130 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.813111 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn" Jul 22 18:57:13.820761 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.820740 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-2nd4q" Jul 22 18:57:13.851633 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.851601 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zmgjp" Jul 22 18:57:13.864881 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:13.864853 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-qtmwq" Jul 22 18:57:14.152291 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:14.152246 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:14.152474 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:14.152449 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 18:57:14.152474 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:14.152472 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c97f75d66-rdvdn: secret "image-registry-tls" not found Jul 22 18:57:14.152597 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:14.152573 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls podName:e07a3932-ff41-4d65-b28d-885e260eec9a nodeName:}" failed. No retries permitted until 2026-07-22 18:57:15.152551062 +0000 UTC m=+38.064715119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls") pod "image-registry-c97f75d66-rdvdn" (UID: "e07a3932-ff41-4d65-b28d-885e260eec9a") : secret "image-registry-tls" not found Jul 22 18:57:14.253145 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:14.253088 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:14.253308 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:14.253160 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9rgk7\" (UID: \"d5899683-5c18-4ee4-9878-2fec416d25a8\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:14.253308 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:14.253216 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:14.253428 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:14.253313 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle podName:dd794227-b4ac-45cb-ab35-ba8f431d0948 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:15.253290969 +0000 UTC m=+38.165455033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle") pod "router-default-8494c9898d-4jsgr" (UID: "dd794227-b4ac-45cb-ab35-ba8f431d0948") : configmap references non-existent config key: service-ca.crt Jul 22 18:57:14.253428 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:14.253322 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 18:57:14.253428 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:14.253350 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 18:57:14.253428 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:14.253365 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls podName:d5899683-5c18-4ee4-9878-2fec416d25a8 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:15.253355353 +0000 UTC m=+38.165519418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9rgk7" (UID: "d5899683-5c18-4ee4-9878-2fec416d25a8") : secret "cluster-monitoring-operator-tls" not found Jul 22 18:57:14.253428 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:14.253400 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 18:57:14.253428 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:14.253411 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls podName:d9d7ca2a-6211-4cf0-af77-49663ce893c3 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:15.253394948 +0000 UTC m=+38.165559027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls") pod "dns-default-dk7bn" (UID: "d9d7ca2a-6211-4cf0-af77-49663ce893c3") : secret "dns-default-metrics-tls" not found Jul 22 18:57:14.253428 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:14.253349 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rq9jp\" (UID: \"a479373f-89ba-4003-81f0-2eb33cb5c35a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" Jul 22 18:57:14.253747 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:14.253435 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls podName:a479373f-89ba-4003-81f0-2eb33cb5c35a nodeName:}" failed. No retries permitted until 2026-07-22 18:57:15.25342473 +0000 UTC m=+38.165588777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-rq9jp" (UID: "a479373f-89ba-4003-81f0-2eb33cb5c35a") : secret "samples-operator-tls" not found Jul 22 18:57:14.253747 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:14.253468 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:14.253747 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:14.253572 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 18:57:14.253747 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:14.253603 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs podName:dd794227-b4ac-45cb-ab35-ba8f431d0948 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:15.253594157 +0000 UTC m=+38.165758204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs") pod "router-default-8494c9898d-4jsgr" (UID: "dd794227-b4ac-45cb-ab35-ba8f431d0948") : secret "router-metrics-certs-default" not found Jul 22 18:57:14.544337 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:14.544302 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:57:14.547300 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:14.547276 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 18:57:15.152774 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.152744 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc"] Jul 22 18:57:15.156956 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:57:15.156923 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d766722_198a_46e6_9175_5c13a9138778.slice/crio-3231e833c53e6ce9639a3dd26dbbd1e10e3c43d46b74b03cf86d21b3a75f882f WatchSource:0}: Error finding container 3231e833c53e6ce9639a3dd26dbbd1e10e3c43d46b74b03cf86d21b3a75f882f: Status 404 returned error can't find the container with id 3231e833c53e6ce9639a3dd26dbbd1e10e3c43d46b74b03cf86d21b3a75f882f Jul 22 18:57:15.164164 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.164127 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-2nd4q"] Jul 22 18:57:15.168918 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:57:15.168889 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0aa0a66_a4d6_4557_bf05_05c34c80c173.slice/crio-532f604f86943e632242f2e3012f65dc3af27ddfcd7eb3ef5b9eef554e9f8814 WatchSource:0}: Error finding container 532f604f86943e632242f2e3012f65dc3af27ddfcd7eb3ef5b9eef554e9f8814: Status 404 returned error can't find the container with id 532f604f86943e632242f2e3012f65dc3af27ddfcd7eb3ef5b9eef554e9f8814 Jul 22 18:57:15.170387 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.170366 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:15.170624 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:15.170562 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 18:57:15.170624 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:15.170582 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c97f75d66-rdvdn: secret "image-registry-tls" not found Jul 22 18:57:15.170729 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:15.170705 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls podName:e07a3932-ff41-4d65-b28d-885e260eec9a nodeName:}" failed. No retries permitted until 2026-07-22 18:57:17.170680918 +0000 UTC m=+40.082844988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls") pod "image-registry-c97f75d66-rdvdn" (UID: "e07a3932-ff41-4d65-b28d-885e260eec9a") : secret "image-registry-tls" not found Jul 22 18:57:15.175172 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.174986 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zmgjp"] Jul 22 18:57:15.177111 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.177084 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-2fmdc"] Jul 22 18:57:15.186581 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:57:15.186554 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2092fe4a_f842_4ebf_876e_3fe8b8ebb0a4.slice/crio-751928b544c15879a61edf0920bca56f7c7f350db1cb3a054eaa5e6eb3bd545c WatchSource:0}: Error finding container 751928b544c15879a61edf0920bca56f7c7f350db1cb3a054eaa5e6eb3bd545c: Status 404 returned error can't find the container with id 751928b544c15879a61edf0920bca56f7c7f350db1cb3a054eaa5e6eb3bd545c Jul 22 18:57:15.186804 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:57:15.186782 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded584a0f_c300_43ea_9cb8_dc928e29f0e8.slice/crio-bbec50f49e6739226907e308129269ba76b8c87bfab78cbf188452805daa26d6 WatchSource:0}: Error finding container bbec50f49e6739226907e308129269ba76b8c87bfab78cbf188452805daa26d6: Status 404 returned error can't find the container with id bbec50f49e6739226907e308129269ba76b8c87bfab78cbf188452805daa26d6 Jul 22 18:57:15.192938 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.192918 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-scnm7"] Jul 22 18:57:15.194464 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.194398 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-qtmwq"] Jul 22 18:57:15.197696 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.197671 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn"] Jul 22 18:57:15.200234 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:57:15.200214 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa91b8fa_def9_4745_b1ec_66df9c2adc5a.slice/crio-6c274f258bdbdb9c1e33418927d053bd288b32fec5e0f7d579854377e1c3314b WatchSource:0}: Error finding container 6c274f258bdbdb9c1e33418927d053bd288b32fec5e0f7d579854377e1c3314b: Status 404 returned error can't find the container with id 6c274f258bdbdb9c1e33418927d053bd288b32fec5e0f7d579854377e1c3314b Jul 22 18:57:15.271624 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.271587 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:15.271775 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.271638 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:15.271775 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.271664 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9rgk7\" (UID: \"d5899683-5c18-4ee4-9878-2fec416d25a8\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:15.271775 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.271721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:15.271775 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.271764 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rq9jp\" (UID: \"a479373f-89ba-4003-81f0-2eb33cb5c35a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" Jul 22 18:57:15.271990 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:15.271781 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 18:57:15.271990 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:15.271796 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle podName:dd794227-b4ac-45cb-ab35-ba8f431d0948 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:17.271775356 +0000 UTC m=+40.183939421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle") pod "router-default-8494c9898d-4jsgr" (UID: "dd794227-b4ac-45cb-ab35-ba8f431d0948") : configmap references non-existent config key: service-ca.crt Jul 22 18:57:15.271990 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:15.271849 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs podName:dd794227-b4ac-45cb-ab35-ba8f431d0948 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:17.271831002 +0000 UTC m=+40.183995070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs") pod "router-default-8494c9898d-4jsgr" (UID: "dd794227-b4ac-45cb-ab35-ba8f431d0948") : secret "router-metrics-certs-default" not found Jul 22 18:57:15.271990 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:15.271850 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 18:57:15.271990 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:15.271858 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 18:57:15.271990 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:15.271865 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 18:57:15.271990 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:15.271892 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls podName:a479373f-89ba-4003-81f0-2eb33cb5c35a nodeName:}" failed. No retries permitted until 2026-07-22 18:57:17.271882771 +0000 UTC m=+40.184046817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-rq9jp" (UID: "a479373f-89ba-4003-81f0-2eb33cb5c35a") : secret "samples-operator-tls" not found Jul 22 18:57:15.271990 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:15.271912 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls podName:d5899683-5c18-4ee4-9878-2fec416d25a8 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:17.271897596 +0000 UTC m=+40.184061678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9rgk7" (UID: "d5899683-5c18-4ee4-9878-2fec416d25a8") : secret "cluster-monitoring-operator-tls" not found Jul 22 18:57:15.271990 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:15.271936 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls podName:d9d7ca2a-6211-4cf0-af77-49663ce893c3 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:17.271924594 +0000 UTC m=+40.184088647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls") pod "dns-default-dk7bn" (UID: "d9d7ca2a-6211-4cf0-af77-49663ce893c3") : secret "dns-default-metrics-tls" not found Jul 22 18:57:15.737435 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.737355 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-qtmwq" event={"ID":"3466f1e1-169b-4031-8f7a-a551062b79eb","Type":"ContainerStarted","Data":"7ff9fad32d6e88c9d572eba6bf55add48ce1c8bd99ac8440a7fe570e8e7498ad"} Jul 22 18:57:15.740509 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.740422 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-2fmdc" event={"ID":"ed584a0f-c300-43ea-9cb8-dc928e29f0e8","Type":"ContainerStarted","Data":"bbec50f49e6739226907e308129269ba76b8c87bfab78cbf188452805daa26d6"} Jul 22 18:57:15.742845 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.742769 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-scnm7" event={"ID":"255f4f29-f31e-4d05-8544-d16d69b75e34","Type":"ContainerStarted","Data":"0ba5b7779712e88b295e52aba9b2e9967658e07140884d64bb81f61e97ed3ffb"} Jul 22 18:57:15.752572 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.752476 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zmgjp" event={"ID":"2092fe4a-f842-4ebf-876e-3fe8b8ebb0a4","Type":"ContainerStarted","Data":"751928b544c15879a61edf0920bca56f7c7f350db1cb3a054eaa5e6eb3bd545c"} Jul 22 18:57:15.758718 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.758628 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc" event={"ID":"1d766722-198a-46e6-9175-5c13a9138778","Type":"ContainerStarted","Data":"3231e833c53e6ce9639a3dd26dbbd1e10e3c43d46b74b03cf86d21b3a75f882f"} Jul 22 18:57:15.763548 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.760697 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn" event={"ID":"aa91b8fa-def9-4745-b1ec-66df9c2adc5a","Type":"ContainerStarted","Data":"6c274f258bdbdb9c1e33418927d053bd288b32fec5e0f7d579854377e1c3314b"} Jul 22 18:57:15.768747 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.768718 2583 generic.go:356] "Generic (PLEG): container finished" podID="85d1fd76-3e08-49cc-8b59-81d1a2c69867" containerID="ab527eb2347db03fb5d65188f9dbf6ce64e1f4b21f93032c50e320a2e24ac444" exitCode=0 Jul 22 18:57:15.768862 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.768790 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6qvj" event={"ID":"85d1fd76-3e08-49cc-8b59-81d1a2c69867","Type":"ContainerDied","Data":"ab527eb2347db03fb5d65188f9dbf6ce64e1f4b21f93032c50e320a2e24ac444"} Jul 22 18:57:15.775800 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:15.775732 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-2nd4q" event={"ID":"e0aa0a66-a4d6-4557-bf05-05c34c80c173","Type":"ContainerStarted","Data":"532f604f86943e632242f2e3012f65dc3af27ddfcd7eb3ef5b9eef554e9f8814"} Jul 22 18:57:16.788295 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:16.787385 2583 generic.go:356] "Generic (PLEG): container finished" podID="85d1fd76-3e08-49cc-8b59-81d1a2c69867" containerID="81481a29bcda80433ba4cb57cc1e2b188f71b29819757246140a561027eb4822" exitCode=0 Jul 22 18:57:16.788295 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:16.787473 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6qvj" event={"ID":"85d1fd76-3e08-49cc-8b59-81d1a2c69867","Type":"ContainerDied","Data":"81481a29bcda80433ba4cb57cc1e2b188f71b29819757246140a561027eb4822"} Jul 22 18:57:17.192605 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:17.192562 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:17.192775 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:17.192734 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 18:57:17.192775 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:17.192757 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c97f75d66-rdvdn: secret "image-registry-tls" not found Jul 22 18:57:17.192899 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:17.192821 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls podName:e07a3932-ff41-4d65-b28d-885e260eec9a nodeName:}" failed. No retries permitted until 2026-07-22 18:57:21.192800016 +0000 UTC m=+44.104964087 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls") pod "image-registry-c97f75d66-rdvdn" (UID: "e07a3932-ff41-4d65-b28d-885e260eec9a") : secret "image-registry-tls" not found Jul 22 18:57:17.294178 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:17.294141 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:17.294343 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:17.294207 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rq9jp\" (UID: \"a479373f-89ba-4003-81f0-2eb33cb5c35a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" Jul 22 18:57:17.294343 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:17.294228 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:17.294343 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:17.294258 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:17.294343 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:17.294286 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9rgk7\" (UID: \"d5899683-5c18-4ee4-9878-2fec416d25a8\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:17.294483 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:17.294390 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 18:57:17.294483 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:17.294437 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls podName:d5899683-5c18-4ee4-9878-2fec416d25a8 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:21.294425045 +0000 UTC m=+44.206589091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9rgk7" (UID: "d5899683-5c18-4ee4-9878-2fec416d25a8") : secret "cluster-monitoring-operator-tls" not found Jul 22 18:57:17.294483 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:17.294476 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 18:57:17.294609 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:17.294520 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls podName:d9d7ca2a-6211-4cf0-af77-49663ce893c3 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:21.294509256 +0000 UTC m=+44.206673303 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls") pod "dns-default-dk7bn" (UID: "d9d7ca2a-6211-4cf0-af77-49663ce893c3") : secret "dns-default-metrics-tls" not found Jul 22 18:57:17.294609 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:17.294564 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 18:57:17.294609 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:17.294590 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls podName:a479373f-89ba-4003-81f0-2eb33cb5c35a nodeName:}" failed. No retries permitted until 2026-07-22 18:57:21.294583416 +0000 UTC m=+44.206747462 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-rq9jp" (UID: "a479373f-89ba-4003-81f0-2eb33cb5c35a") : secret "samples-operator-tls" not found Jul 22 18:57:17.294724 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:17.294634 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 18:57:17.294724 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:17.294661 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs podName:dd794227-b4ac-45cb-ab35-ba8f431d0948 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:21.294651498 +0000 UTC m=+44.206815545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs") pod "router-default-8494c9898d-4jsgr" (UID: "dd794227-b4ac-45cb-ab35-ba8f431d0948") : secret "router-metrics-certs-default" not found Jul 22 18:57:17.294724 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:17.294713 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle podName:dd794227-b4ac-45cb-ab35-ba8f431d0948 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:21.294706267 +0000 UTC m=+44.206870313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle") pod "router-default-8494c9898d-4jsgr" (UID: "dd794227-b4ac-45cb-ab35-ba8f431d0948") : configmap references non-existent config key: service-ca.crt Jul 22 18:57:17.796833 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:17.796041 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6qvj" event={"ID":"85d1fd76-3e08-49cc-8b59-81d1a2c69867","Type":"ContainerStarted","Data":"d7d63c0f4e6b6969b514bc2433eb84046e2e8fe283c061400ee459a376493040"} Jul 22 18:57:17.823274 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:17.823221 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t6qvj" podStartSLOduration=8.695025813 podStartE2EDuration="39.823205938s" totalImagesPullingTime="31.128180125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:56:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:17.822564345 +0000 UTC m=+40.734728413" watchObservedRunningTime="2026-07-22 18:57:17.823205938 +0000 UTC m=+40.735370035" Jul 22 18:57:21.227948 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:21.227909 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:21.228470 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:21.228120 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 18:57:21.228470 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:21.228140 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c97f75d66-rdvdn: secret "image-registry-tls" not found Jul 22 18:57:21.228470 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:21.228202 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls podName:e07a3932-ff41-4d65-b28d-885e260eec9a nodeName:}" failed. No retries permitted until 2026-07-22 18:57:29.228184482 +0000 UTC m=+52.140348528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls") pod "image-registry-c97f75d66-rdvdn" (UID: "e07a3932-ff41-4d65-b28d-885e260eec9a") : secret "image-registry-tls" not found Jul 22 18:57:21.328344 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:21.328298 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9rgk7\" (UID: \"d5899683-5c18-4ee4-9878-2fec416d25a8\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:21.328568 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:21.328381 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:21.328568 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:21.328430 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rq9jp\" (UID: \"a479373f-89ba-4003-81f0-2eb33cb5c35a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" Jul 22 18:57:21.328568 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:21.328451 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:21.328568 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:21.328511 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:21.328568 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:21.328529 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 18:57:21.328842 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:21.328588 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 18:57:21.328842 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:21.328529 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 18:57:21.328842 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:21.328595 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls podName:d9d7ca2a-6211-4cf0-af77-49663ce893c3 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:29.328580988 +0000 UTC m=+52.240745049 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls") pod "dns-default-dk7bn" (UID: "d9d7ca2a-6211-4cf0-af77-49663ce893c3") : secret "dns-default-metrics-tls" not found Jul 22 18:57:21.328842 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:21.328656 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle podName:dd794227-b4ac-45cb-ab35-ba8f431d0948 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:29.328639682 +0000 UTC m=+52.240803740 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle") pod "router-default-8494c9898d-4jsgr" (UID: "dd794227-b4ac-45cb-ab35-ba8f431d0948") : configmap references non-existent config key: service-ca.crt Jul 22 18:57:21.328842 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:21.328660 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 18:57:21.328842 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:21.328669 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls podName:a479373f-89ba-4003-81f0-2eb33cb5c35a nodeName:}" failed. No retries permitted until 2026-07-22 18:57:29.328662376 +0000 UTC m=+52.240826421 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-rq9jp" (UID: "a479373f-89ba-4003-81f0-2eb33cb5c35a") : secret "samples-operator-tls" not found Jul 22 18:57:21.328842 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:21.328685 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls podName:d5899683-5c18-4ee4-9878-2fec416d25a8 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:29.32868025 +0000 UTC m=+52.240844297 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9rgk7" (UID: "d5899683-5c18-4ee4-9878-2fec416d25a8") : secret "cluster-monitoring-operator-tls" not found Jul 22 18:57:21.328842 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:21.328725 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs podName:dd794227-b4ac-45cb-ab35-ba8f431d0948 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:29.328713953 +0000 UTC m=+52.240877998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs") pod "router-default-8494c9898d-4jsgr" (UID: "dd794227-b4ac-45cb-ab35-ba8f431d0948") : secret "router-metrics-certs-default" not found Jul 22 18:57:23.814810 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:23.814770 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zmgjp" event={"ID":"2092fe4a-f842-4ebf-876e-3fe8b8ebb0a4","Type":"ContainerStarted","Data":"e37759facacb6b9b99eef5ecc9da7b645c7d5a0c922e730d1f5d702a77a36128"} Jul 22 18:57:23.817113 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:23.816705 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc" event={"ID":"1d766722-198a-46e6-9175-5c13a9138778","Type":"ContainerStarted","Data":"d30a775c542c665058f5436251433948631b2f9887f8d996ffd211025a618c27"} Jul 22 18:57:23.818483 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:23.818440 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn" event={"ID":"aa91b8fa-def9-4745-b1ec-66df9c2adc5a","Type":"ContainerStarted","Data":"14a0ac7986dfeb13070f375b8d64a29d1f7b4cf9d27c931ff2d74f872fa1017d"} Jul 22 18:57:23.820385 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:23.820183 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-2nd4q" event={"ID":"e0aa0a66-a4d6-4557-bf05-05c34c80c173","Type":"ContainerStarted","Data":"c9f2cd033a80146753be47c360dc24a95f9eb905e4ed8c5c1747720a7b458313"} Jul 22 18:57:23.821737 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:23.821714 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-qtmwq" event={"ID":"3466f1e1-169b-4031-8f7a-a551062b79eb","Type":"ContainerStarted","Data":"2945199b7e21d1ffaac908b742d195241165a7437c20348860a4e34fa71cf1da"} Jul 22 18:57:23.823215 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:23.823174 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-2fmdc" event={"ID":"ed584a0f-c300-43ea-9cb8-dc928e29f0e8","Type":"ContainerStarted","Data":"632b899f3a42bd42296e1799c8b334c5480684cbd35ef4c0f3955f9bd6c9c40e"} Jul 22 18:57:23.824826 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:23.824807 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/0.log" Jul 22 18:57:23.824926 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:23.824844 2583 generic.go:356] "Generic (PLEG): container finished" podID="255f4f29-f31e-4d05-8544-d16d69b75e34" containerID="9e63e3af9dd9a8329a36da57c71e75f3216b9d469bde6a6efdc0b0f8f5c424c6" exitCode=255 Jul 22 18:57:23.824926 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:23.824873 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-scnm7" event={"ID":"255f4f29-f31e-4d05-8544-d16d69b75e34","Type":"ContainerDied","Data":"9e63e3af9dd9a8329a36da57c71e75f3216b9d469bde6a6efdc0b0f8f5c424c6"} Jul 22 18:57:23.825154 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:23.825135 2583 scope.go:117] "RemoveContainer" containerID="9e63e3af9dd9a8329a36da57c71e75f3216b9d469bde6a6efdc0b0f8f5c424c6" Jul 22 18:57:23.830908 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:23.830811 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zmgjp" podStartSLOduration=2.958652416 podStartE2EDuration="10.830799503s" totalImagesPullingTime="7.872147087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:23.830119022 +0000 UTC m=+46.742283090" watchObservedRunningTime="2026-07-22 18:57:23.830799503 +0000 UTC m=+46.742963571" Jul 22 18:57:23.882568 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:23.879464 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-qtmwq" podStartSLOduration=4.017591212 podStartE2EDuration="11.87944284s" totalImagesPullingTime="7.861851628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:23.850009603 +0000 UTC m=+46.762173673" watchObservedRunningTime="2026-07-22 18:57:23.87944284 +0000 UTC m=+46.791606910" Jul 22 18:57:23.901617 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:23.901563 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-2fmdc" podStartSLOduration=4.162938457 podStartE2EDuration="11.901540366s" totalImagesPullingTime="7.738601909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:23.899409984 +0000 UTC m=+46.811574053" watchObservedRunningTime="2026-07-22 18:57:23.901540366 +0000 UTC m=+46.813704414" Jul 22 18:57:23.916029 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:23.915977 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc" podStartSLOduration=4.017313614 podStartE2EDuration="11.91595815s" totalImagesPullingTime="7.898644536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:23.915340726 +0000 UTC m=+46.827504795" watchObservedRunningTime="2026-07-22 18:57:23.91595815 +0000 UTC m=+46.828122219" Jul 22 18:57:23.934604 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:23.934537 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn" podStartSLOduration=4.071556086 podStartE2EDuration="11.934517642s" totalImagesPullingTime="7.862961556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:23.93270075 +0000 UTC m=+46.844864825" watchObservedRunningTime="2026-07-22 18:57:23.934517642 +0000 UTC m=+46.846681703" Jul 22 18:57:24.410068 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.406902 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-2nd4q" podStartSLOduration=4.51726046 podStartE2EDuration="12.406864866s" totalImagesPullingTime="7.889604406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:23.956410457 +0000 UTC m=+46.868574525" watchObservedRunningTime="2026-07-22 18:57:24.406864866 +0000 UTC m=+47.319028937" Jul 22 18:57:24.410068 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.408426 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bxp4d"] Jul 22 18:57:24.424622 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.423970 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:24.427401 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.427088 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 18:57:24.427401 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.427152 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 18:57:24.427401 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.427090 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wspht\"" Jul 22 18:57:24.428726 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.428087 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bxp4d"] Jul 22 18:57:24.458533 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.458476 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/300c2416-b0a7-4abc-93c9-da7234dd1d2e-crio-socket\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:24.458703 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.458587 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/300c2416-b0a7-4abc-93c9-da7234dd1d2e-data-volume\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:24.458703 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.458625 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:24.458821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.458709 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djhtd\" (UniqueName: \"kubernetes.io/projected/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-api-access-djhtd\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:24.458821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.458760 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:24.559894 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.559855 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhtd\" (UniqueName: \"kubernetes.io/projected/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-api-access-djhtd\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:24.560051 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.559923 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:24.560051 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.559989 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/300c2416-b0a7-4abc-93c9-da7234dd1d2e-crio-socket\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:24.560153 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.560050 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/300c2416-b0a7-4abc-93c9-da7234dd1d2e-data-volume\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:24.560153 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:24.560056 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 18:57:24.560153 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.560089 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:24.560153 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:24.560128 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-rbac-proxy-cm podName:300c2416-b0a7-4abc-93c9-da7234dd1d2e nodeName:}" failed. No retries permitted until 2026-07-22 18:57:25.060106246 +0000 UTC m=+47.972270297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bxp4d" (UID: "300c2416-b0a7-4abc-93c9-da7234dd1d2e") : configmap "kube-rbac-proxy" not found Jul 22 18:57:24.560577 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.560188 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/300c2416-b0a7-4abc-93c9-da7234dd1d2e-crio-socket\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:24.560577 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:24.560435 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 18:57:24.560577 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.560442 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/300c2416-b0a7-4abc-93c9-da7234dd1d2e-data-volume\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:24.560577 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:24.560529 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls podName:300c2416-b0a7-4abc-93c9-da7234dd1d2e nodeName:}" failed. No retries permitted until 2026-07-22 18:57:25.06047983 +0000 UTC m=+47.972643882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bxp4d" (UID: "300c2416-b0a7-4abc-93c9-da7234dd1d2e") : secret "insights-runtime-extractor-tls" not found Jul 22 18:57:24.575164 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.575136 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djhtd\" (UniqueName: \"kubernetes.io/projected/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-api-access-djhtd\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:24.830288 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.830257 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 18:57:24.830783 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.830658 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/0.log" Jul 22 18:57:24.830783 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.830689 2583 generic.go:356] "Generic (PLEG): container finished" podID="255f4f29-f31e-4d05-8544-d16d69b75e34" containerID="901434c4cb0429398a4abf91ccbaaa9933e6c354bcdde20503b0f8a283e3b4c4" exitCode=255 Jul 22 18:57:24.830783 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.830765 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-scnm7" event={"ID":"255f4f29-f31e-4d05-8544-d16d69b75e34","Type":"ContainerDied","Data":"901434c4cb0429398a4abf91ccbaaa9933e6c354bcdde20503b0f8a283e3b4c4"} Jul 22 18:57:24.830934 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.830814 2583 scope.go:117] "RemoveContainer" containerID="9e63e3af9dd9a8329a36da57c71e75f3216b9d469bde6a6efdc0b0f8f5c424c6" Jul 22 18:57:24.831046 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:24.831023 2583 scope.go:117] "RemoveContainer" containerID="901434c4cb0429398a4abf91ccbaaa9933e6c354bcdde20503b0f8a283e3b4c4" Jul 22 18:57:24.831238 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:24.831213 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-scnm7_openshift-console-operator(255f4f29-f31e-4d05-8544-d16d69b75e34)\"" pod="openshift-console-operator/console-operator-565b7b569d-scnm7" podUID="255f4f29-f31e-4d05-8544-d16d69b75e34" Jul 22 18:57:25.065279 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:25.065243 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:25.065445 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:25.065343 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:25.065445 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:25.065416 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 18:57:25.065445 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:25.065438 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 18:57:25.065628 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:25.065522 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls podName:300c2416-b0a7-4abc-93c9-da7234dd1d2e nodeName:}" failed. No retries permitted until 2026-07-22 18:57:26.065476248 +0000 UTC m=+48.977640317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bxp4d" (UID: "300c2416-b0a7-4abc-93c9-da7234dd1d2e") : secret "insights-runtime-extractor-tls" not found Jul 22 18:57:25.065628 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:25.065548 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-rbac-proxy-cm podName:300c2416-b0a7-4abc-93c9-da7234dd1d2e nodeName:}" failed. No retries permitted until 2026-07-22 18:57:26.065535216 +0000 UTC m=+48.977699263 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bxp4d" (UID: "300c2416-b0a7-4abc-93c9-da7234dd1d2e") : configmap "kube-rbac-proxy" not found Jul 22 18:57:25.836073 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:25.836046 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 18:57:25.836441 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:25.836367 2583 scope.go:117] "RemoveContainer" containerID="901434c4cb0429398a4abf91ccbaaa9933e6c354bcdde20503b0f8a283e3b4c4" Jul 22 18:57:25.836555 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:25.836536 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-scnm7_openshift-console-operator(255f4f29-f31e-4d05-8544-d16d69b75e34)\"" pod="openshift-console-operator/console-operator-565b7b569d-scnm7" podUID="255f4f29-f31e-4d05-8544-d16d69b75e34" Jul 22 18:57:26.074380 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:26.074336 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:26.074600 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:26.074434 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:26.074600 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:26.074530 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 18:57:26.074600 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:26.074586 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 18:57:26.074744 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:26.074608 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls podName:300c2416-b0a7-4abc-93c9-da7234dd1d2e nodeName:}" failed. No retries permitted until 2026-07-22 18:57:28.074588247 +0000 UTC m=+50.986752310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bxp4d" (UID: "300c2416-b0a7-4abc-93c9-da7234dd1d2e") : secret "insights-runtime-extractor-tls" not found Jul 22 18:57:26.074744 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:26.074651 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-rbac-proxy-cm podName:300c2416-b0a7-4abc-93c9-da7234dd1d2e nodeName:}" failed. No retries permitted until 2026-07-22 18:57:28.074632682 +0000 UTC m=+50.986796742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bxp4d" (UID: "300c2416-b0a7-4abc-93c9-da7234dd1d2e") : configmap "kube-rbac-proxy" not found Jul 22 18:57:26.212232 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:26.212153 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2fd99_0961d651-b746-4c2e-85f5-d2a00c044e3a/dns-node-resolver/0.log" Jul 22 18:57:27.411663 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:27.411634 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fxb82_00c3d60d-32a0-41ce-9ce6-45b51a3dd7da/node-ca/0.log" Jul 22 18:57:28.091223 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:28.091182 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:28.091422 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:28.091301 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:28.091422 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:28.091330 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 18:57:28.091422 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:28.091405 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-rbac-proxy-cm podName:300c2416-b0a7-4abc-93c9-da7234dd1d2e nodeName:}" failed. No retries permitted until 2026-07-22 18:57:32.091385914 +0000 UTC m=+55.003549984 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bxp4d" (UID: "300c2416-b0a7-4abc-93c9-da7234dd1d2e") : configmap "kube-rbac-proxy" not found Jul 22 18:57:28.091581 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:28.091442 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 18:57:28.091581 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:28.091485 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls podName:300c2416-b0a7-4abc-93c9-da7234dd1d2e nodeName:}" failed. No retries permitted until 2026-07-22 18:57:32.091474037 +0000 UTC m=+55.003638085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bxp4d" (UID: "300c2416-b0a7-4abc-93c9-da7234dd1d2e") : secret "insights-runtime-extractor-tls" not found Jul 22 18:57:28.413430 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:28.413361 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zmgjp_2092fe4a-f842-4ebf-876e-3fe8b8ebb0a4/serve-healthcheck-canary/0.log" Jul 22 18:57:29.301268 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:29.301229 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:29.301443 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:29.301337 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 18:57:29.301443 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:29.301350 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c97f75d66-rdvdn: secret "image-registry-tls" not found Jul 22 18:57:29.301443 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:29.301397 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls podName:e07a3932-ff41-4d65-b28d-885e260eec9a nodeName:}" failed. No retries permitted until 2026-07-22 18:57:45.301383811 +0000 UTC m=+68.213547857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls") pod "image-registry-c97f75d66-rdvdn" (UID: "e07a3932-ff41-4d65-b28d-885e260eec9a") : secret "image-registry-tls" not found Jul 22 18:57:29.402019 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:29.401980 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:29.402184 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:29.402027 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9rgk7\" (UID: \"d5899683-5c18-4ee4-9878-2fec416d25a8\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:29.402184 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:29.402088 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:29.402184 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:29.402151 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle podName:dd794227-b4ac-45cb-ab35-ba8f431d0948 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:45.402133523 +0000 UTC m=+68.314297570 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle") pod "router-default-8494c9898d-4jsgr" (UID: "dd794227-b4ac-45cb-ab35-ba8f431d0948") : configmap references non-existent config key: service-ca.crt Jul 22 18:57:29.402314 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:29.402196 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rq9jp\" (UID: \"a479373f-89ba-4003-81f0-2eb33cb5c35a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" Jul 22 18:57:29.402314 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:29.402209 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 18:57:29.402314 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:29.402220 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:29.402314 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:29.402263 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 18:57:29.402314 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:29.402209 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 18:57:29.402314 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:29.402265 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls podName:d5899683-5c18-4ee4-9878-2fec416d25a8 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:45.402247839 +0000 UTC m=+68.314411899 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9rgk7" (UID: "d5899683-5c18-4ee4-9878-2fec416d25a8") : secret "cluster-monitoring-operator-tls" not found Jul 22 18:57:29.402314 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:29.402310 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls podName:a479373f-89ba-4003-81f0-2eb33cb5c35a nodeName:}" failed. No retries permitted until 2026-07-22 18:57:45.402299067 +0000 UTC m=+68.314463115 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-rq9jp" (UID: "a479373f-89ba-4003-81f0-2eb33cb5c35a") : secret "samples-operator-tls" not found Jul 22 18:57:29.402314 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:29.402310 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 18:57:29.402611 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:29.402328 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls podName:d9d7ca2a-6211-4cf0-af77-49663ce893c3 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:45.402322243 +0000 UTC m=+68.314486289 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls") pod "dns-default-dk7bn" (UID: "d9d7ca2a-6211-4cf0-af77-49663ce893c3") : secret "dns-default-metrics-tls" not found Jul 22 18:57:29.402611 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:29.402348 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs podName:dd794227-b4ac-45cb-ab35-ba8f431d0948 nodeName:}" failed. No retries permitted until 2026-07-22 18:57:45.402340004 +0000 UTC m=+68.314504049 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs") pod "router-default-8494c9898d-4jsgr" (UID: "dd794227-b4ac-45cb-ab35-ba8f431d0948") : secret "router-metrics-certs-default" not found Jul 22 18:57:32.124051 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:32.124011 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:32.124708 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:32.124139 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:32.124708 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:32.124171 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 18:57:32.124708 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:32.124239 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls podName:300c2416-b0a7-4abc-93c9-da7234dd1d2e nodeName:}" failed. No retries permitted until 2026-07-22 18:57:40.124220272 +0000 UTC m=+63.036384346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bxp4d" (UID: "300c2416-b0a7-4abc-93c9-da7234dd1d2e") : secret "insights-runtime-extractor-tls" not found Jul 22 18:57:32.124890 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:32.124861 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/300c2416-b0a7-4abc-93c9-da7234dd1d2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:33.805157 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:33.805120 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:33.805576 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:33.805180 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:33.805576 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:33.805566 2583 scope.go:117] "RemoveContainer" containerID="901434c4cb0429398a4abf91ccbaaa9933e6c354bcdde20503b0f8a283e3b4c4" Jul 22 18:57:33.806162 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:57:33.805712 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-scnm7_openshift-console-operator(255f4f29-f31e-4d05-8544-d16d69b75e34)\"" pod="openshift-console-operator/console-operator-565b7b569d-scnm7" podUID="255f4f29-f31e-4d05-8544-d16d69b75e34" Jul 22 18:57:38.728634 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:38.728605 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66v5g" Jul 22 18:57:40.194969 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:40.194930 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:40.197335 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:40.197314 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/300c2416-b0a7-4abc-93c9-da7234dd1d2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bxp4d\" (UID: \"300c2416-b0a7-4abc-93c9-da7234dd1d2e\") " pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:40.339383 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:40.339350 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wspht\"" Jul 22 18:57:40.347520 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:40.347480 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bxp4d" Jul 22 18:57:40.474293 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:40.474212 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bxp4d"] Jul 22 18:57:40.476962 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:57:40.476932 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300c2416_b0a7_4abc_93c9_da7234dd1d2e.slice/crio-acfd3bfc8a2fb381750f66f4864565946d8d6e25b0e8257485adf828c2de9d47 WatchSource:0}: Error finding container acfd3bfc8a2fb381750f66f4864565946d8d6e25b0e8257485adf828c2de9d47: Status 404 returned error can't find the container with id acfd3bfc8a2fb381750f66f4864565946d8d6e25b0e8257485adf828c2de9d47 Jul 22 18:57:40.890362 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:40.890314 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bxp4d" event={"ID":"300c2416-b0a7-4abc-93c9-da7234dd1d2e","Type":"ContainerStarted","Data":"acfd3bfc8a2fb381750f66f4864565946d8d6e25b0e8257485adf828c2de9d47"} Jul 22 18:57:43.226639 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:43.226601 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret\") pod \"global-pull-secret-syncer-j79dm\" (UID: \"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd\") " pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:57:43.229523 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:43.229487 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 18:57:43.240329 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:43.240301 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ac11ffa-90fb-40e6-a021-0984d4d8c9bd-original-pull-secret\") pod \"global-pull-secret-syncer-j79dm\" (UID: \"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd\") " pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:57:43.355311 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:43.355273 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j79dm" Jul 22 18:57:43.427933 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:43.427891 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs\") pod \"network-metrics-daemon-p27ms\" (UID: \"abb50681-d581-483f-a16a-bccdb437e24e\") " pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:57:43.428110 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:43.427957 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wx9g\" (UniqueName: \"kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g\") pod \"network-check-target-bdzlg\" (UID: \"e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc\") " pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:57:43.430994 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:43.430965 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 18:57:43.431137 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:43.431117 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wx9g\" (UniqueName: \"kubernetes.io/projected/e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc-kube-api-access-6wx9g\") pod \"network-check-target-bdzlg\" (UID: \"e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc\") " pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:57:43.441283 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:43.441259 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb50681-d581-483f-a16a-bccdb437e24e-metrics-certs\") pod \"network-metrics-daemon-p27ms\" (UID: \"abb50681-d581-483f-a16a-bccdb437e24e\") " pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:57:43.575148 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:43.575118 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5knzw\"" Jul 22 18:57:43.580528 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:43.580480 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g8nmg\"" Jul 22 18:57:43.582555 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:43.582534 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p27ms" Jul 22 18:57:43.588262 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:43.588240 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:57:45.344267 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.343919 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:45.346548 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.346528 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls\") pod \"image-registry-c97f75d66-rdvdn\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:45.392133 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.392106 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bdzlg"] Jul 22 18:57:45.398626 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:57:45.398603 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ea95f8_b9aa_4eb8_af27_4ba52bc085fc.slice/crio-901b6e98f796ce4060b98c77d9fa5c63a31855975891c0ca36eabee613cd12cf WatchSource:0}: Error finding container 901b6e98f796ce4060b98c77d9fa5c63a31855975891c0ca36eabee613cd12cf: Status 404 returned error can't find the container with id 901b6e98f796ce4060b98c77d9fa5c63a31855975891c0ca36eabee613cd12cf Jul 22 18:57:45.403879 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.403856 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j79dm"] Jul 22 18:57:45.406452 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:57:45.406429 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac11ffa_90fb_40e6_a021_0984d4d8c9bd.slice/crio-ebf8ef7f23307329e9f621ec5cecaacabab2142590ff3ab68f0c1578f0a77358 WatchSource:0}: Error finding container ebf8ef7f23307329e9f621ec5cecaacabab2142590ff3ab68f0c1578f0a77358: Status 404 returned error can't find the container with id ebf8ef7f23307329e9f621ec5cecaacabab2142590ff3ab68f0c1578f0a77358 Jul 22 18:57:45.433382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.433357 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p27ms"] Jul 22 18:57:45.445193 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.445172 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:45.445277 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.445216 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rq9jp\" (UID: \"a479373f-89ba-4003-81f0-2eb33cb5c35a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" Jul 22 18:57:45.445277 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.445235 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:45.445277 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.445260 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:45.445419 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.445284 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9rgk7\" (UID: \"d5899683-5c18-4ee4-9878-2fec416d25a8\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:45.446449 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.446025 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd794227-b4ac-45cb-ab35-ba8f431d0948-service-ca-bundle\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:45.447909 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.447871 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9d7ca2a-6211-4cf0-af77-49663ce893c3-metrics-tls\") pod \"dns-default-dk7bn\" (UID: \"d9d7ca2a-6211-4cf0-af77-49663ce893c3\") " pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:45.448278 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.448251 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5899683-5c18-4ee4-9878-2fec416d25a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9rgk7\" (UID: \"d5899683-5c18-4ee4-9878-2fec416d25a8\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:45.448825 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.448734 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a479373f-89ba-4003-81f0-2eb33cb5c35a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rq9jp\" (UID: \"a479373f-89ba-4003-81f0-2eb33cb5c35a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" Jul 22 18:57:45.448901 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.448834 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd794227-b4ac-45cb-ab35-ba8f431d0948-metrics-certs\") pod \"router-default-8494c9898d-4jsgr\" (UID: \"dd794227-b4ac-45cb-ab35-ba8f431d0948\") " pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:45.560988 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.560956 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bx5gq\"" Jul 22 18:57:45.569135 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.569110 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:45.631473 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.631450 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zs4n4\"" Jul 22 18:57:45.638923 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.638897 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" Jul 22 18:57:45.639380 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.639358 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-w2dk4\"" Jul 22 18:57:45.646658 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.646637 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mqrp5\"" Jul 22 18:57:45.647680 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.647664 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:45.654592 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.654547 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" Jul 22 18:57:45.662681 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.662655 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cc98d\"" Jul 22 18:57:45.669593 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.669566 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:45.714097 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.713989 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c97f75d66-rdvdn"] Jul 22 18:57:45.717070 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:57:45.717027 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07a3932_ff41_4d65_b28d_885e260eec9a.slice/crio-add5a24224b0b04645c0594b7565a099e81b18e847e335d337d82b57b2c89735 WatchSource:0}: Error finding container add5a24224b0b04645c0594b7565a099e81b18e847e335d337d82b57b2c89735: Status 404 returned error can't find the container with id add5a24224b0b04645c0594b7565a099e81b18e847e335d337d82b57b2c89735 Jul 22 18:57:45.847043 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.846970 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8494c9898d-4jsgr"] Jul 22 18:57:45.851660 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:57:45.851561 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd794227_b4ac_45cb_ab35_ba8f431d0948.slice/crio-b0a104c1233afeec10985433c5d7f92a1cf7006339e3f7a8518750484910d54e WatchSource:0}: Error finding container b0a104c1233afeec10985433c5d7f92a1cf7006339e3f7a8518750484910d54e: Status 404 returned error can't find the container with id b0a104c1233afeec10985433c5d7f92a1cf7006339e3f7a8518750484910d54e Jul 22 18:57:45.851660 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.851602 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp"] Jul 22 18:57:45.871343 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.871309 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7"] Jul 22 18:57:45.901869 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.901844 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dk7bn"] Jul 22 18:57:45.905703 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:57:45.905675 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d7ca2a_6211_4cf0_af77_49663ce893c3.slice/crio-a07ac5591abcb09a761353f3071d4827cb0b1888ac964e5a1c7c42f907419029 WatchSource:0}: Error finding container a07ac5591abcb09a761353f3071d4827cb0b1888ac964e5a1c7c42f907419029: Status 404 returned error can't find the container with id a07ac5591abcb09a761353f3071d4827cb0b1888ac964e5a1c7c42f907419029 Jul 22 18:57:45.910331 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.910302 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" event={"ID":"e07a3932-ff41-4d65-b28d-885e260eec9a","Type":"ContainerStarted","Data":"570703a13cdd1166e4dad23d9e71ea61099a43c90c0d9b8c97f5822cd4b8a189"} Jul 22 18:57:45.910414 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.910361 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" event={"ID":"e07a3932-ff41-4d65-b28d-885e260eec9a","Type":"ContainerStarted","Data":"add5a24224b0b04645c0594b7565a099e81b18e847e335d337d82b57b2c89735"} Jul 22 18:57:45.910844 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.910799 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:57:45.913055 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.913031 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bxp4d" event={"ID":"300c2416-b0a7-4abc-93c9-da7234dd1d2e","Type":"ContainerStarted","Data":"319c2b269fa74b7784d2cac5166799fc0b227ba9f3997a771a3e0ad414833db8"} Jul 22 18:57:45.914602 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.914572 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" event={"ID":"d5899683-5c18-4ee4-9878-2fec416d25a8","Type":"ContainerStarted","Data":"8b4b5758799f33921c4a382ae76f2ead50ffde72f5cbcda54e7af75333daef4c"} Jul 22 18:57:45.917013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.916990 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8494c9898d-4jsgr" event={"ID":"dd794227-b4ac-45cb-ab35-ba8f431d0948","Type":"ContainerStarted","Data":"b0a104c1233afeec10985433c5d7f92a1cf7006339e3f7a8518750484910d54e"} Jul 22 18:57:45.918726 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.918695 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bdzlg" event={"ID":"e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc","Type":"ContainerStarted","Data":"a54b45e0e6bd2e225ce70ccc1658e087681667825b99348ec5e17199ae59361b"} Jul 22 18:57:45.918809 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.918735 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bdzlg" event={"ID":"e0ea95f8-b9aa-4eb8-af27-4ba52bc085fc","Type":"ContainerStarted","Data":"901b6e98f796ce4060b98c77d9fa5c63a31855975891c0ca36eabee613cd12cf"} Jul 22 18:57:45.918858 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.918842 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:57:45.920284 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.920263 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j79dm" event={"ID":"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd","Type":"ContainerStarted","Data":"ebf8ef7f23307329e9f621ec5cecaacabab2142590ff3ab68f0c1578f0a77358"} Jul 22 18:57:45.921378 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.921356 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p27ms" event={"ID":"abb50681-d581-483f-a16a-bccdb437e24e","Type":"ContainerStarted","Data":"c4125d209251b5f05a84dfe00bc43fe48ba9b31fbc6756a39fb55aed3a904f5c"} Jul 22 18:57:45.955204 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.955150 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" podStartSLOduration=44.955126707 podStartE2EDuration="44.955126707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:45.938151954 +0000 UTC m=+68.850316024" watchObservedRunningTime="2026-07-22 18:57:45.955126707 +0000 UTC m=+68.867290775" Jul 22 18:57:45.957163 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:45.957110 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bdzlg" podStartSLOduration=67.957092807 podStartE2EDuration="1m7.957092807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:45.95588279 +0000 UTC m=+68.868046857" watchObservedRunningTime="2026-07-22 18:57:45.957092807 +0000 UTC m=+68.869256872" Jul 22 18:57:46.928404 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:46.928361 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8494c9898d-4jsgr" event={"ID":"dd794227-b4ac-45cb-ab35-ba8f431d0948","Type":"ContainerStarted","Data":"95f3f2676a71a9e8ce9988081a9c327878d93089281683c300271e68836c2737"} Jul 22 18:57:46.929687 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:46.929654 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dk7bn" event={"ID":"d9d7ca2a-6211-4cf0-af77-49663ce893c3","Type":"ContainerStarted","Data":"a07ac5591abcb09a761353f3071d4827cb0b1888ac964e5a1c7c42f907419029"} Jul 22 18:57:46.932375 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:46.932348 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" event={"ID":"a479373f-89ba-4003-81f0-2eb33cb5c35a","Type":"ContainerStarted","Data":"be832498351f104a82979a84c0eaf33910528f05f7d187e3efc2b29d57d3d92e"} Jul 22 18:57:46.952330 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:46.952273 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-8494c9898d-4jsgr" podStartSLOduration=34.952252942 podStartE2EDuration="34.952252942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:46.950293685 +0000 UTC m=+69.862457754" watchObservedRunningTime="2026-07-22 18:57:46.952252942 +0000 UTC m=+69.864417009" Jul 22 18:57:47.547223 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:47.546748 2583 scope.go:117] "RemoveContainer" containerID="901434c4cb0429398a4abf91ccbaaa9933e6c354bcdde20503b0f8a283e3b4c4" Jul 22 18:57:47.649148 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:47.649070 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:47.651357 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:47.651138 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:47.947310 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:47.944707 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 18:57:47.947310 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:47.944856 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-scnm7" event={"ID":"255f4f29-f31e-4d05-8544-d16d69b75e34","Type":"ContainerStarted","Data":"80de01c8fa469d76697136e2aa689eaa86f6c3655416fd57e35e2cc28e9805fe"} Jul 22 18:57:47.948986 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:47.948924 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:47.956208 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:47.956109 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p27ms" event={"ID":"abb50681-d581-483f-a16a-bccdb437e24e","Type":"ContainerStarted","Data":"3258d958d7bf729bae58b94089d8977765f4602d964256c616676887ac3d2af1"} Jul 22 18:57:47.956208 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:47.956154 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p27ms" event={"ID":"abb50681-d581-483f-a16a-bccdb437e24e","Type":"ContainerStarted","Data":"8c315d604d6fc76c6e991893b6e7b02b2a4316e28b137738d2f8558ee08152f2"} Jul 22 18:57:47.956208 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:47.956179 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:47.958093 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:47.958054 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8494c9898d-4jsgr" Jul 22 18:57:47.965294 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:47.965238 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-scnm7" podStartSLOduration=28.105709613 podStartE2EDuration="35.965220228s" totalImagesPullingTime="7.859510615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:47.964692044 +0000 UTC m=+70.876856126" watchObservedRunningTime="2026-07-22 18:57:47.965220228 +0000 UTC m=+70.877384298" Jul 22 18:57:47.999573 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:47.998219 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p27ms" podStartSLOduration=68.452247294 podStartE2EDuration="1m9.998198296s" totalImagesPullingTime="1.545951002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:47.997665599 +0000 UTC m=+70.909829692" watchObservedRunningTime="2026-07-22 18:57:47.998198296 +0000 UTC m=+70.910362368" Jul 22 18:57:48.637841 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:48.637808 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-scnm7" Jul 22 18:57:54.987170 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:54.987103 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" event={"ID":"d5899683-5c18-4ee4-9878-2fec416d25a8","Type":"ContainerStarted","Data":"83ea6305a5d8d932bee2d3c6568c9b91f2e6314da528b3b5597928d3b8fc7f49"} Jul 22 18:57:54.989510 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:54.989428 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j79dm" event={"ID":"5ac11ffa-90fb-40e6-a021-0984d4d8c9bd","Type":"ContainerStarted","Data":"8a01a6a4ec9a6ff98249d306e8f09d58e4367c1907bd766e90de27d571d41a9d"} Jul 22 18:57:54.991533 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:54.991239 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dk7bn" event={"ID":"d9d7ca2a-6211-4cf0-af77-49663ce893c3","Type":"ContainerStarted","Data":"d635edda932d5e8f172b043c7a1153a47b685699d99d6b4580f3511e59079e14"} Jul 22 18:57:54.993520 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:54.993478 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" event={"ID":"a479373f-89ba-4003-81f0-2eb33cb5c35a","Type":"ContainerStarted","Data":"38a1eb5f2d150b4dba7a39c553cb0076dc6101186372d317176e58f7f1d13501"} Jul 22 18:57:54.993597 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:54.993546 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" event={"ID":"a479373f-89ba-4003-81f0-2eb33cb5c35a","Type":"ContainerStarted","Data":"5abefa56cc62e099d1a985c52c260d870c3e1d9279fe51416987ed82c1ddff72"} Jul 22 18:57:55.008479 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:55.008430 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9rgk7" podStartSLOduration=34.401521147 podStartE2EDuration="43.008411971s" totalImagesPullingTime="8.606890824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:55.00834033 +0000 UTC m=+77.920504399" watchObservedRunningTime="2026-07-22 18:57:55.008411971 +0000 UTC m=+77.920576040" Jul 22 18:57:55.034341 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:55.032637 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j79dm" podStartSLOduration=67.956168678 podStartE2EDuration="1m17.032615694s" totalImagesPullingTime="9.076447016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:55.032111379 +0000 UTC m=+77.944275442" watchObservedRunningTime="2026-07-22 18:57:55.032615694 +0000 UTC m=+77.944779763" Jul 22 18:57:55.051877 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:55.051814 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rq9jp" podStartSLOduration=34.501558767 podStartE2EDuration="43.051793448s" totalImagesPullingTime="8.550234681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:55.049900612 +0000 UTC m=+77.962064683" watchObservedRunningTime="2026-07-22 18:57:55.051793448 +0000 UTC m=+77.963957515" Jul 22 18:57:55.999801 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:55.999755 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bxp4d" event={"ID":"300c2416-b0a7-4abc-93c9-da7234dd1d2e","Type":"ContainerStarted","Data":"ca2f97c8a65b1580dad34fbffaf79af031f8f7f3fcf42dbe53bbc6da4c2eb0c1"} Jul 22 18:57:56.001933 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:56.001904 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dk7bn" event={"ID":"d9d7ca2a-6211-4cf0-af77-49663ce893c3","Type":"ContainerStarted","Data":"d946006100a7ff0a3c47d6fb205cee86ee5f98706acd477c60e4877e454cecc5"} Jul 22 18:57:56.002087 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:56.001977 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dk7bn" Jul 22 18:57:56.020370 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:56.020324 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dk7bn" podStartSLOduration=34.442697441 podStartE2EDuration="43.020306361s" totalImagesPullingTime="8.57760892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:56.019249838 +0000 UTC m=+78.931413907" watchObservedRunningTime="2026-07-22 18:57:56.020306361 +0000 UTC m=+78.932470428" Jul 22 18:57:59.015260 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:59.015164 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bxp4d" event={"ID":"300c2416-b0a7-4abc-93c9-da7234dd1d2e","Type":"ContainerStarted","Data":"7660af269de7481aca0dc4d7248f0c9708782721325184e3129860222e2e4031"} Jul 22 18:57:59.033652 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:57:59.033485 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bxp4d" podStartSLOduration=17.197648643 podStartE2EDuration="35.0334707s" totalImagesPullingTime="17.835822057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:57:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:57:59.033437044 +0000 UTC m=+81.945601112" watchObservedRunningTime="2026-07-22 18:57:59.0334707 +0000 UTC m=+81.945634767" Jul 22 18:58:05.574082 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:05.574046 2583 patch_prober.go:28] interesting pod/image-registry-c97f75d66-rdvdn container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 22 18:58:05.574482 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:05.574110 2583 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" podUID="e07a3932-ff41-4d65-b28d-885e260eec9a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 18:58:06.007922 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:06.007841 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dk7bn" Jul 22 18:58:06.936761 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:06.936734 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:58:07.489074 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.489039 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf"] Jul 22 18:58:07.491818 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.491800 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" Jul 22 18:58:07.494720 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.494695 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dksq9\"" Jul 22 18:58:07.494846 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.494727 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 22 18:58:07.494846 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.494701 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 22 18:58:07.495467 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.495200 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 18:58:07.503329 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.503304 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf"] Jul 22 18:58:07.517315 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.517284 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-279fz"] Jul 22 18:58:07.519580 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.519563 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.522308 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.522288 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 22 18:58:07.522441 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.522327 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-46skg\"" Jul 22 18:58:07.522441 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.522341 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 22 18:58:07.522441 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.522363 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 22 18:58:07.538170 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.538141 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-279fz"] Jul 22 18:58:07.543799 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.543773 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p2d9d"] Jul 22 18:58:07.546126 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.546104 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.548965 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.548941 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 18:58:07.549196 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.549165 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fvp2b\"" Jul 22 18:58:07.549316 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.549232 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 18:58:07.549476 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.549454 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 18:58:07.626882 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.626852 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlsdl\" (UniqueName: \"kubernetes.io/projected/8be1063f-28df-4a01-9546-23b7de7547ff-kube-api-access-wlsdl\") pod \"openshift-state-metrics-79f6958cdf-lc8nf\" (UID: \"8be1063f-28df-4a01-9546-23b7de7547ff\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" Jul 22 18:58:07.627056 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.626892 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.627056 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.626912 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tshxc\" (UniqueName: \"kubernetes.io/projected/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-kube-api-access-tshxc\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.627056 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.626938 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8be1063f-28df-4a01-9546-23b7de7547ff-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-lc8nf\" (UID: \"8be1063f-28df-4a01-9546-23b7de7547ff\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" Jul 22 18:58:07.627056 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.626965 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.627056 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.626986 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8be1063f-28df-4a01-9546-23b7de7547ff-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-lc8nf\" (UID: \"8be1063f-28df-4a01-9546-23b7de7547ff\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" Jul 22 18:58:07.627056 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.627008 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.627284 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.627076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.627284 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.627149 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.627284 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.627176 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8be1063f-28df-4a01-9546-23b7de7547ff-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-lc8nf\" (UID: \"8be1063f-28df-4a01-9546-23b7de7547ff\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" Jul 22 18:58:07.727559 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.727469 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8be1063f-28df-4a01-9546-23b7de7547ff-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-lc8nf\" (UID: \"8be1063f-28df-4a01-9546-23b7de7547ff\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" Jul 22 18:58:07.727743 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.727572 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.727743 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.727604 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.727743 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.727637 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/171ffb8e-7b10-4e0a-bef2-299ecff632ef-node-exporter-textfile\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.727743 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.727690 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.727743 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.727713 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/171ffb8e-7b10-4e0a-bef2-299ecff632ef-sys\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.727743 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.727737 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/171ffb8e-7b10-4e0a-bef2-299ecff632ef-node-exporter-tls\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.728043 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.727772 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8be1063f-28df-4a01-9546-23b7de7547ff-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-lc8nf\" (UID: \"8be1063f-28df-4a01-9546-23b7de7547ff\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" Jul 22 18:58:07.728043 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.727803 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67pnp\" (UniqueName: \"kubernetes.io/projected/171ffb8e-7b10-4e0a-bef2-299ecff632ef-kube-api-access-67pnp\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.728043 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.727834 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlsdl\" (UniqueName: \"kubernetes.io/projected/8be1063f-28df-4a01-9546-23b7de7547ff-kube-api-access-wlsdl\") pod \"openshift-state-metrics-79f6958cdf-lc8nf\" (UID: \"8be1063f-28df-4a01-9546-23b7de7547ff\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" Jul 22 18:58:07.728043 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.727858 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/171ffb8e-7b10-4e0a-bef2-299ecff632ef-node-exporter-wtmp\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.728043 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.727887 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/171ffb8e-7b10-4e0a-bef2-299ecff632ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.728043 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.727923 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.728043 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.727950 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tshxc\" (UniqueName: \"kubernetes.io/projected/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-kube-api-access-tshxc\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.728043 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.727975 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/171ffb8e-7b10-4e0a-bef2-299ecff632ef-root\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.728043 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.728020 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/171ffb8e-7b10-4e0a-bef2-299ecff632ef-metrics-client-ca\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.728459 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.728052 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8be1063f-28df-4a01-9546-23b7de7547ff-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-lc8nf\" (UID: \"8be1063f-28df-4a01-9546-23b7de7547ff\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" Jul 22 18:58:07.728459 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.728072 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.728459 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.728080 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/171ffb8e-7b10-4e0a-bef2-299ecff632ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.728459 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.728134 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.728692 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.728558 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8be1063f-28df-4a01-9546-23b7de7547ff-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-lc8nf\" (UID: \"8be1063f-28df-4a01-9546-23b7de7547ff\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" Jul 22 18:58:07.729105 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.729059 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.729613 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.729587 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.730693 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.730673 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.730789 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.730773 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.730837 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.730808 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8be1063f-28df-4a01-9546-23b7de7547ff-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-lc8nf\" (UID: \"8be1063f-28df-4a01-9546-23b7de7547ff\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" Jul 22 18:58:07.731063 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.731047 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8be1063f-28df-4a01-9546-23b7de7547ff-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-lc8nf\" (UID: \"8be1063f-28df-4a01-9546-23b7de7547ff\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" Jul 22 18:58:07.739723 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.739657 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlsdl\" (UniqueName: \"kubernetes.io/projected/8be1063f-28df-4a01-9546-23b7de7547ff-kube-api-access-wlsdl\") pod \"openshift-state-metrics-79f6958cdf-lc8nf\" (UID: \"8be1063f-28df-4a01-9546-23b7de7547ff\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" Jul 22 18:58:07.741916 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.741896 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tshxc\" (UniqueName: \"kubernetes.io/projected/0ec28774-2cd9-4851-8907-6e8bd22a7b3b-kube-api-access-tshxc\") pod \"kube-state-metrics-56c5594674-279fz\" (UID: \"0ec28774-2cd9-4851-8907-6e8bd22a7b3b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.805801 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.805767 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" Jul 22 18:58:07.828930 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.828896 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/171ffb8e-7b10-4e0a-bef2-299ecff632ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.829107 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.828948 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/171ffb8e-7b10-4e0a-bef2-299ecff632ef-root\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.829107 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.828979 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/171ffb8e-7b10-4e0a-bef2-299ecff632ef-metrics-client-ca\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.829107 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.829010 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/171ffb8e-7b10-4e0a-bef2-299ecff632ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.829107 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.829071 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/171ffb8e-7b10-4e0a-bef2-299ecff632ef-node-exporter-textfile\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.829344 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.829117 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/171ffb8e-7b10-4e0a-bef2-299ecff632ef-sys\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.829344 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.829142 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/171ffb8e-7b10-4e0a-bef2-299ecff632ef-node-exporter-tls\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.829344 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.829205 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67pnp\" (UniqueName: \"kubernetes.io/projected/171ffb8e-7b10-4e0a-bef2-299ecff632ef-kube-api-access-67pnp\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.829344 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.829292 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/171ffb8e-7b10-4e0a-bef2-299ecff632ef-node-exporter-wtmp\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.829577 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.829516 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/171ffb8e-7b10-4e0a-bef2-299ecff632ef-node-exporter-wtmp\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.829577 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.829554 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/171ffb8e-7b10-4e0a-bef2-299ecff632ef-root\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.829704 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.829688 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" Jul 22 18:58:07.829885 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.829849 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/171ffb8e-7b10-4e0a-bef2-299ecff632ef-node-exporter-textfile\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.830349 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.830176 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/171ffb8e-7b10-4e0a-bef2-299ecff632ef-sys\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.830562 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.830536 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/171ffb8e-7b10-4e0a-bef2-299ecff632ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.830720 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.830625 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/171ffb8e-7b10-4e0a-bef2-299ecff632ef-metrics-client-ca\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.833213 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.833178 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/171ffb8e-7b10-4e0a-bef2-299ecff632ef-node-exporter-tls\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.833315 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.833219 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/171ffb8e-7b10-4e0a-bef2-299ecff632ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.840321 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.840288 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67pnp\" (UniqueName: \"kubernetes.io/projected/171ffb8e-7b10-4e0a-bef2-299ecff632ef-kube-api-access-67pnp\") pod \"node-exporter-p2d9d\" (UID: \"171ffb8e-7b10-4e0a-bef2-299ecff632ef\") " pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.855240 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.855206 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p2d9d" Jul 22 18:58:07.869299 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:58:07.869255 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171ffb8e_7b10_4e0a_bef2_299ecff632ef.slice/crio-3a6a6f9b516ad365007405614aa046712b20fb31c321590d56520e5840619b83 WatchSource:0}: Error finding container 3a6a6f9b516ad365007405614aa046712b20fb31c321590d56520e5840619b83: Status 404 returned error can't find the container with id 3a6a6f9b516ad365007405614aa046712b20fb31c321590d56520e5840619b83 Jul 22 18:58:07.950755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.950724 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf"] Jul 22 18:58:07.954006 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:58:07.953974 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be1063f_28df_4a01_9546_23b7de7547ff.slice/crio-6da5f3f3834ded7b4e42426f97c3930bf889bb4d2adae6b09214b602fd9eb974 WatchSource:0}: Error finding container 6da5f3f3834ded7b4e42426f97c3930bf889bb4d2adae6b09214b602fd9eb974: Status 404 returned error can't find the container with id 6da5f3f3834ded7b4e42426f97c3930bf889bb4d2adae6b09214b602fd9eb974 Jul 22 18:58:07.978017 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:07.977995 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-279fz"] Jul 22 18:58:07.981569 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:58:07.981538 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec28774_2cd9_4851_8907_6e8bd22a7b3b.slice/crio-d0cb40afd8b69e6938ba4e24a1aa2d92a6813a6b29fd88c348ca0ba48380bfef WatchSource:0}: Error finding container d0cb40afd8b69e6938ba4e24a1aa2d92a6813a6b29fd88c348ca0ba48380bfef: Status 404 returned error can't find the container with id d0cb40afd8b69e6938ba4e24a1aa2d92a6813a6b29fd88c348ca0ba48380bfef Jul 22 18:58:08.046371 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.046335 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" event={"ID":"0ec28774-2cd9-4851-8907-6e8bd22a7b3b","Type":"ContainerStarted","Data":"d0cb40afd8b69e6938ba4e24a1aa2d92a6813a6b29fd88c348ca0ba48380bfef"} Jul 22 18:58:08.047561 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.047533 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p2d9d" event={"ID":"171ffb8e-7b10-4e0a-bef2-299ecff632ef","Type":"ContainerStarted","Data":"3a6a6f9b516ad365007405614aa046712b20fb31c321590d56520e5840619b83"} Jul 22 18:58:08.049476 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.049451 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" event={"ID":"8be1063f-28df-4a01-9546-23b7de7547ff","Type":"ContainerStarted","Data":"6722c2ba3c109a8fa93f40fc4468a9353ecccad0c0331ef1f4680a76ebbcfc43"} Jul 22 18:58:08.049597 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.049523 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" event={"ID":"8be1063f-28df-4a01-9546-23b7de7547ff","Type":"ContainerStarted","Data":"6da5f3f3834ded7b4e42426f97c3930bf889bb4d2adae6b09214b602fd9eb974"} Jul 22 18:58:08.603716 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.597069 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 18:58:08.608414 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.607230 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.610383 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.610351 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 18:58:08.610779 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.610760 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 18:58:08.610992 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.610946 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 18:58:08.611162 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.611140 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 18:58:08.611458 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.611409 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kh8sb\"" Jul 22 18:58:08.611627 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.611603 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 18:58:08.612530 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.611762 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 18:58:08.612530 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.611829 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 18:58:08.612530 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.612174 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 18:58:08.612530 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.612239 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 18:58:08.618990 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.618939 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 18:58:08.736883 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.736848 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.737039 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.736901 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.737039 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.736940 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/48a1a2ca-a41a-4c23-b65e-30264abdf11d-config-out\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.737039 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.736967 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-config-volume\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.737039 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.736996 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48a1a2ca-a41a-4c23-b65e-30264abdf11d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.737039 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.737022 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.737258 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.737045 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.737258 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.737068 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7pt8\" (UniqueName: \"kubernetes.io/projected/48a1a2ca-a41a-4c23-b65e-30264abdf11d-kube-api-access-w7pt8\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.737258 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.737127 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48a1a2ca-a41a-4c23-b65e-30264abdf11d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.737258 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.737152 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/48a1a2ca-a41a-4c23-b65e-30264abdf11d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.737258 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.737175 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/48a1a2ca-a41a-4c23-b65e-30264abdf11d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.737258 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.737226 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.737258 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.737249 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-web-config\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.837955 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.837924 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-config-volume\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.838076 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.837968 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48a1a2ca-a41a-4c23-b65e-30264abdf11d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.838076 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.837996 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.838076 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.838023 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.838076 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.838045 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7pt8\" (UniqueName: \"kubernetes.io/projected/48a1a2ca-a41a-4c23-b65e-30264abdf11d-kube-api-access-w7pt8\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.838314 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.838111 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48a1a2ca-a41a-4c23-b65e-30264abdf11d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.838314 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.838137 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/48a1a2ca-a41a-4c23-b65e-30264abdf11d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.838314 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.838160 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/48a1a2ca-a41a-4c23-b65e-30264abdf11d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.838314 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.838204 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.838314 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.838230 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-web-config\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.838314 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.838262 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.838314 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.838302 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.838660 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.838341 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/48a1a2ca-a41a-4c23-b65e-30264abdf11d-config-out\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.838660 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:58:08.838625 2583 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 22 18:58:08.838762 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:58:08.838695 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-main-tls podName:48a1a2ca-a41a-4c23-b65e-30264abdf11d nodeName:}" failed. No retries permitted until 2026-07-22 18:58:09.33867244 +0000 UTC m=+92.250836500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "48a1a2ca-a41a-4c23-b65e-30264abdf11d") : secret "alertmanager-main-tls" not found Jul 22 18:58:08.839665 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.839544 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48a1a2ca-a41a-4c23-b65e-30264abdf11d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.840691 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.839760 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48a1a2ca-a41a-4c23-b65e-30264abdf11d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.840691 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.840627 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/48a1a2ca-a41a-4c23-b65e-30264abdf11d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.841568 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.841487 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/48a1a2ca-a41a-4c23-b65e-30264abdf11d-config-out\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.842648 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.842627 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.844025 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.843881 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/48a1a2ca-a41a-4c23-b65e-30264abdf11d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.845866 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.845806 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.845866 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.845861 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.846645 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.846276 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-config-volume\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.846645 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.846602 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.849057 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.849012 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-web-config\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:08.849335 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:08.849316 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7pt8\" (UniqueName: \"kubernetes.io/projected/48a1a2ca-a41a-4c23-b65e-30264abdf11d-kube-api-access-w7pt8\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:09.057348 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:09.057256 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" event={"ID":"8be1063f-28df-4a01-9546-23b7de7547ff","Type":"ContainerStarted","Data":"17e004c11c887de55b4e164f1e57c5335abd6b518fe21a358ebf8a7d66593ac9"} Jul 22 18:58:09.059712 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:09.059684 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p2d9d" event={"ID":"171ffb8e-7b10-4e0a-bef2-299ecff632ef","Type":"ContainerStarted","Data":"40f68469142b9bfcd9fc8f6f87ade72fa6158d092783f55bfae77d10e6368ed9"} Jul 22 18:58:09.342007 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:09.341967 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:09.344788 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:09.344754 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:09.522115 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:09.522089 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:58:09.708101 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:09.708073 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 18:58:10.065611 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:10.065569 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" event={"ID":"8be1063f-28df-4a01-9546-23b7de7547ff","Type":"ContainerStarted","Data":"d687a283c5abfa9f61731e1bb729efaa3aad01743eb0542f8c4b58e669620ebb"} Jul 22 18:58:10.067746 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:10.067720 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" event={"ID":"0ec28774-2cd9-4851-8907-6e8bd22a7b3b","Type":"ContainerStarted","Data":"c2eb5b936a04567c85918d3d2c39edfcf14308ddf90f5ba4e8d4eaf1daeb16e2"} Jul 22 18:58:10.067882 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:10.067774 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" event={"ID":"0ec28774-2cd9-4851-8907-6e8bd22a7b3b","Type":"ContainerStarted","Data":"baa6a9435861abab6325cc6f8f97bdf7d734d479108eb0e93c1fc3d9b95a412e"} Jul 22 18:58:10.067882 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:10.067802 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" event={"ID":"0ec28774-2cd9-4851-8907-6e8bd22a7b3b","Type":"ContainerStarted","Data":"bd573c5ccf138b17a8e0d80011c772daf9f383bdbc45a2b749e8909cc04539d6"} Jul 22 18:58:10.069036 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:10.068998 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48a1a2ca-a41a-4c23-b65e-30264abdf11d","Type":"ContainerStarted","Data":"3b1e12d80d7b03c0819822604e41467d6f6f296f2e00448484152e00e4dc1fb1"} Jul 22 18:58:10.070875 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:10.070840 2583 generic.go:356] "Generic (PLEG): container finished" podID="171ffb8e-7b10-4e0a-bef2-299ecff632ef" containerID="40f68469142b9bfcd9fc8f6f87ade72fa6158d092783f55bfae77d10e6368ed9" exitCode=0 Jul 22 18:58:10.070975 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:10.070905 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p2d9d" event={"ID":"171ffb8e-7b10-4e0a-bef2-299ecff632ef","Type":"ContainerDied","Data":"40f68469142b9bfcd9fc8f6f87ade72fa6158d092783f55bfae77d10e6368ed9"} Jul 22 18:58:10.087645 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:10.087602 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-lc8nf" podStartSLOduration=1.684830017 podStartE2EDuration="3.087586186s" totalImagesPullingTime="1.402756169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:58:10.086327118 +0000 UTC m=+92.998491185" watchObservedRunningTime="2026-07-22 18:58:10.087586186 +0000 UTC m=+92.999750254" Jul 22 18:58:10.128805 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:10.128746 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-279fz" podStartSLOduration=1.6258177489999999 podStartE2EDuration="3.12872978s" totalImagesPullingTime="1.502912031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:58:10.127214672 +0000 UTC m=+93.039378741" watchObservedRunningTime="2026-07-22 18:58:10.12872978 +0000 UTC m=+93.040893847" Jul 22 18:58:11.076839 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:11.076802 2583 generic.go:356] "Generic (PLEG): container finished" podID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerID="547e447da426e717c1712e7dd4d3a0734188ce793abc645f7c39aa1497d34ae9" exitCode=0 Jul 22 18:58:11.077261 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:11.076889 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48a1a2ca-a41a-4c23-b65e-30264abdf11d","Type":"ContainerDied","Data":"547e447da426e717c1712e7dd4d3a0734188ce793abc645f7c39aa1497d34ae9"} Jul 22 18:58:11.079532 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:11.079479 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p2d9d" event={"ID":"171ffb8e-7b10-4e0a-bef2-299ecff632ef","Type":"ContainerStarted","Data":"4b6c3d7bcf254a625bfb837181f6091561aaff25bd3b41e6e6fd1f073d6669e1"} Jul 22 18:58:11.079628 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:11.079563 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p2d9d" event={"ID":"171ffb8e-7b10-4e0a-bef2-299ecff632ef","Type":"ContainerStarted","Data":"7f8b6b26ceb10ca7dc6df09e5a537df478d2a6f032477cbd292b24129af18416"} Jul 22 18:58:11.130321 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:11.130270 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p2d9d" podStartSLOduration=3.086901779 podStartE2EDuration="4.130253856s" totalImagesPullingTime="956.162085ms" totalInitContainerRuntime="87.189992ms" isStatefulPod=true podCreationTimestamp="2026-07-22 18:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:58:11.129140731 +0000 UTC m=+94.041304800" watchObservedRunningTime="2026-07-22 18:58:11.130253856 +0000 UTC m=+94.042417923" Jul 22 18:58:12.286525 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.286481 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2cmfw"] Jul 22 18:58:12.289411 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.289383 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2cmfw" Jul 22 18:58:12.292003 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.291982 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 22 18:58:12.292129 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.292008 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pnwbr\"" Jul 22 18:58:12.298847 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.298825 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2cmfw"] Jul 22 18:58:12.470420 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.470385 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8782b139-b8d9-4ef9-877a-d68684d267f2-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-2cmfw\" (UID: \"8782b139-b8d9-4ef9-877a-d68684d267f2\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2cmfw" Jul 22 18:58:12.470619 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.470447 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b2q2\" (UniqueName: \"kubernetes.io/projected/8782b139-b8d9-4ef9-877a-d68684d267f2-kube-api-access-4b2q2\") pod \"monitoring-plugin-7b7b7bc8bf-2cmfw\" (UID: \"8782b139-b8d9-4ef9-877a-d68684d267f2\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2cmfw" Jul 22 18:58:12.561961 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.561891 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55c7887b67-fph6t"] Jul 22 18:58:12.563917 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.563901 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.566449 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.566423 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 18:58:12.566559 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.566541 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 18:58:12.566749 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.566734 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 18:58:12.566810 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.566740 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 18:58:12.566866 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.566777 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5r47k\"" Jul 22 18:58:12.566988 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.566973 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 18:58:12.567382 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.567369 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 18:58:12.567450 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.567432 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 18:58:12.571156 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.570964 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8782b139-b8d9-4ef9-877a-d68684d267f2-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-2cmfw\" (UID: \"8782b139-b8d9-4ef9-877a-d68684d267f2\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2cmfw" Jul 22 18:58:12.571156 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.571010 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b2q2\" (UniqueName: \"kubernetes.io/projected/8782b139-b8d9-4ef9-877a-d68684d267f2-kube-api-access-4b2q2\") pod \"monitoring-plugin-7b7b7bc8bf-2cmfw\" (UID: \"8782b139-b8d9-4ef9-877a-d68684d267f2\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2cmfw" Jul 22 18:58:12.578558 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.574699 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 18:58:12.578558 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.574923 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8782b139-b8d9-4ef9-877a-d68684d267f2-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-2cmfw\" (UID: \"8782b139-b8d9-4ef9-877a-d68684d267f2\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2cmfw" Jul 22 18:58:12.578558 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.577834 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55c7887b67-fph6t"] Jul 22 18:58:12.584692 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.584666 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b2q2\" (UniqueName: \"kubernetes.io/projected/8782b139-b8d9-4ef9-877a-d68684d267f2-kube-api-access-4b2q2\") pod \"monitoring-plugin-7b7b7bc8bf-2cmfw\" (UID: \"8782b139-b8d9-4ef9-877a-d68684d267f2\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2cmfw" Jul 22 18:58:12.600561 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.600537 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2cmfw" Jul 22 18:58:12.673065 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.672385 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0334e485-8398-417e-834e-0def6853ae0d-console-oauth-config\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.673065 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.672513 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-console-config\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.673065 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.672548 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-oauth-serving-cert\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.673065 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.672573 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sg2n\" (UniqueName: \"kubernetes.io/projected/0334e485-8398-417e-834e-0def6853ae0d-kube-api-access-7sg2n\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.673065 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.672599 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0334e485-8398-417e-834e-0def6853ae0d-console-serving-cert\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.673065 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.672650 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-trusted-ca-bundle\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.673065 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.672676 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-service-ca\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.750526 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.750474 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-59f988c449-c5x4p"] Jul 22 18:58:12.753782 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.753750 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.756202 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.756180 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 22 18:58:12.756620 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.756598 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 22 18:58:12.756729 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.756712 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 22 18:58:12.756884 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.756869 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 22 18:58:12.756952 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.756938 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 22 18:58:12.757178 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.757163 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2dzwt\"" Jul 22 18:58:12.761396 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.761375 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 22 18:58:12.770449 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.770423 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-59f988c449-c5x4p"] Jul 22 18:58:12.773469 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.773446 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2cmfw"] Jul 22 18:58:12.774037 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.774008 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sg2n\" (UniqueName: \"kubernetes.io/projected/0334e485-8398-417e-834e-0def6853ae0d-kube-api-access-7sg2n\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.774142 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.774060 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0334e485-8398-417e-834e-0def6853ae0d-console-serving-cert\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.774142 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.774116 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-trusted-ca-bundle\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.774372 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.774144 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-service-ca\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.774372 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.774178 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0334e485-8398-417e-834e-0def6853ae0d-console-oauth-config\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.774372 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.774314 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-console-config\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.775272 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.774946 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-oauth-serving-cert\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.775583 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.775562 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-service-ca\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.775800 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.775779 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-console-config\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.775953 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.775934 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-trusted-ca-bundle\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.776221 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.776200 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-oauth-serving-cert\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.777234 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:58:12.777202 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8782b139_b8d9_4ef9_877a_d68684d267f2.slice/crio-d631762df5ebb6ac50e735090612c01048ad12cab7916f9760eb3f13c2c6ec45 WatchSource:0}: Error finding container d631762df5ebb6ac50e735090612c01048ad12cab7916f9760eb3f13c2c6ec45: Status 404 returned error can't find the container with id d631762df5ebb6ac50e735090612c01048ad12cab7916f9760eb3f13c2c6ec45 Jul 22 18:58:12.778526 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.778483 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0334e485-8398-417e-834e-0def6853ae0d-console-serving-cert\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.779768 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.779745 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0334e485-8398-417e-834e-0def6853ae0d-console-oauth-config\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.782401 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.782380 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sg2n\" (UniqueName: \"kubernetes.io/projected/0334e485-8398-417e-834e-0def6853ae0d-kube-api-access-7sg2n\") pod \"console-55c7887b67-fph6t\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.875060 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.875037 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:12.875402 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.875383 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59b62c85-911a-4519-ae05-e0f9a2d71907-metrics-client-ca\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.875458 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.875415 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/59b62c85-911a-4519-ae05-e0f9a2d71907-secret-telemeter-client\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.875458 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.875437 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/59b62c85-911a-4519-ae05-e0f9a2d71907-telemeter-client-tls\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.875631 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.875457 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59b62c85-911a-4519-ae05-e0f9a2d71907-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.875631 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.875478 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b62c85-911a-4519-ae05-e0f9a2d71907-serving-certs-ca-bundle\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.875631 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.875585 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/59b62c85-911a-4519-ae05-e0f9a2d71907-federate-client-tls\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.875631 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.875608 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz6m4\" (UniqueName: \"kubernetes.io/projected/59b62c85-911a-4519-ae05-e0f9a2d71907-kube-api-access-tz6m4\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.875631 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.875626 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b62c85-911a-4519-ae05-e0f9a2d71907-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.977181 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.976790 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59b62c85-911a-4519-ae05-e0f9a2d71907-metrics-client-ca\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.977181 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.976841 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/59b62c85-911a-4519-ae05-e0f9a2d71907-secret-telemeter-client\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.977181 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.976877 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/59b62c85-911a-4519-ae05-e0f9a2d71907-telemeter-client-tls\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.977181 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.976909 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59b62c85-911a-4519-ae05-e0f9a2d71907-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.977181 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.976944 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b62c85-911a-4519-ae05-e0f9a2d71907-serving-certs-ca-bundle\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.977181 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.976996 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/59b62c85-911a-4519-ae05-e0f9a2d71907-federate-client-tls\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.977181 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.977020 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz6m4\" (UniqueName: \"kubernetes.io/projected/59b62c85-911a-4519-ae05-e0f9a2d71907-kube-api-access-tz6m4\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.977181 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.977051 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b62c85-911a-4519-ae05-e0f9a2d71907-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.978204 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.978015 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b62c85-911a-4519-ae05-e0f9a2d71907-serving-certs-ca-bundle\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.978204 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.978173 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b62c85-911a-4519-ae05-e0f9a2d71907-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.978393 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.978342 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59b62c85-911a-4519-ae05-e0f9a2d71907-metrics-client-ca\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.980909 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.980852 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/59b62c85-911a-4519-ae05-e0f9a2d71907-telemeter-client-tls\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.981919 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.981620 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/59b62c85-911a-4519-ae05-e0f9a2d71907-secret-telemeter-client\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.982024 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.981999 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59b62c85-911a-4519-ae05-e0f9a2d71907-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.982383 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.982349 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/59b62c85-911a-4519-ae05-e0f9a2d71907-federate-client-tls\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:12.985990 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:12.985963 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz6m4\" (UniqueName: \"kubernetes.io/projected/59b62c85-911a-4519-ae05-e0f9a2d71907-kube-api-access-tz6m4\") pod \"telemeter-client-59f988c449-c5x4p\" (UID: \"59b62c85-911a-4519-ae05-e0f9a2d71907\") " pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:13.023446 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:13.023418 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55c7887b67-fph6t"] Jul 22 18:58:13.025520 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:58:13.025468 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0334e485_8398_417e_834e_0def6853ae0d.slice/crio-2402816420fa9f3e835a875084e73d09f869704b7b7a45f5c7871950372b4b19 WatchSource:0}: Error finding container 2402816420fa9f3e835a875084e73d09f869704b7b7a45f5c7871950372b4b19: Status 404 returned error can't find the container with id 2402816420fa9f3e835a875084e73d09f869704b7b7a45f5c7871950372b4b19 Jul 22 18:58:13.078603 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:13.078481 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" Jul 22 18:58:13.091388 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:13.091363 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48a1a2ca-a41a-4c23-b65e-30264abdf11d","Type":"ContainerStarted","Data":"07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a"} Jul 22 18:58:13.091539 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:13.091428 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48a1a2ca-a41a-4c23-b65e-30264abdf11d","Type":"ContainerStarted","Data":"79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667"} Jul 22 18:58:13.091539 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:13.091454 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48a1a2ca-a41a-4c23-b65e-30264abdf11d","Type":"ContainerStarted","Data":"c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd"} Jul 22 18:58:13.091539 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:13.091506 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48a1a2ca-a41a-4c23-b65e-30264abdf11d","Type":"ContainerStarted","Data":"b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326"} Jul 22 18:58:13.091677 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:13.091548 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48a1a2ca-a41a-4c23-b65e-30264abdf11d","Type":"ContainerStarted","Data":"ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b"} Jul 22 18:58:13.092629 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:13.092606 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2cmfw" event={"ID":"8782b139-b8d9-4ef9-877a-d68684d267f2","Type":"ContainerStarted","Data":"d631762df5ebb6ac50e735090612c01048ad12cab7916f9760eb3f13c2c6ec45"} Jul 22 18:58:13.093868 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:13.093844 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55c7887b67-fph6t" event={"ID":"0334e485-8398-417e-834e-0def6853ae0d","Type":"ContainerStarted","Data":"2402816420fa9f3e835a875084e73d09f869704b7b7a45f5c7871950372b4b19"} Jul 22 18:58:13.212624 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:13.212598 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-59f988c449-c5x4p"] Jul 22 18:58:13.214365 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:58:13.214336 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b62c85_911a_4519_ae05_e0f9a2d71907.slice/crio-3f2aec99479a41e3a54915d32ea74c403d116d46fcb0faa1c8cff236aacde85a WatchSource:0}: Error finding container 3f2aec99479a41e3a54915d32ea74c403d116d46fcb0faa1c8cff236aacde85a: Status 404 returned error can't find the container with id 3f2aec99479a41e3a54915d32ea74c403d116d46fcb0faa1c8cff236aacde85a Jul 22 18:58:14.111205 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:14.111165 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48a1a2ca-a41a-4c23-b65e-30264abdf11d","Type":"ContainerStarted","Data":"e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba"} Jul 22 18:58:14.114260 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:14.114190 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" event={"ID":"59b62c85-911a-4519-ae05-e0f9a2d71907","Type":"ContainerStarted","Data":"3f2aec99479a41e3a54915d32ea74c403d116d46fcb0faa1c8cff236aacde85a"} Jul 22 18:58:14.142197 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:14.142134 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.9765236120000003 podStartE2EDuration="6.142111596s" totalImagesPullingTime="3.150250513s" totalInitContainerRuntime="15.337471ms" isStatefulPod=false podCreationTimestamp="2026-07-22 18:58:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:58:14.141761239 +0000 UTC m=+97.053925333" watchObservedRunningTime="2026-07-22 18:58:14.142111596 +0000 UTC m=+97.054275666" Jul 22 18:58:15.120914 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:15.120876 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2cmfw" event={"ID":"8782b139-b8d9-4ef9-877a-d68684d267f2","Type":"ContainerStarted","Data":"fd364f91dc7be35658cab87a3272ad98ec93d3634c81a9ee993bcdef7b4ace62"} Jul 22 18:58:15.121541 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:15.121456 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2cmfw" Jul 22 18:58:15.127145 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:15.127121 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2cmfw" Jul 22 18:58:15.140403 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:15.140353 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2cmfw" podStartSLOduration=1.268056739 podStartE2EDuration="3.140336871s" totalImagesPullingTime="1.872280132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:58:15.138297767 +0000 UTC m=+98.050461848" watchObservedRunningTime="2026-07-22 18:58:15.140336871 +0000 UTC m=+98.052500939" Jul 22 18:58:16.485079 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:16.485044 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c97f75d66-rdvdn"] Jul 22 18:58:16.935796 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:16.935775 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bdzlg" Jul 22 18:58:17.136831 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:17.136794 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55c7887b67-fph6t" event={"ID":"0334e485-8398-417e-834e-0def6853ae0d","Type":"ContainerStarted","Data":"ecb0268cb4d2322f8e16a9b06500d152df6ebc33266f94689287190fe45f0614"} Jul 22 18:58:17.139038 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:17.139012 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" event={"ID":"59b62c85-911a-4519-ae05-e0f9a2d71907","Type":"ContainerStarted","Data":"170379a118f1c029bb52c1d4b09b34199f3a02aa38314ab0d006874d5991778c"} Jul 22 18:58:17.139147 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:17.139056 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" event={"ID":"59b62c85-911a-4519-ae05-e0f9a2d71907","Type":"ContainerStarted","Data":"874dbb904580e309c58e1ccacad7467b75f4154938f42e1ef86221c045142d97"} Jul 22 18:58:17.139147 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:17.139080 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" event={"ID":"59b62c85-911a-4519-ae05-e0f9a2d71907","Type":"ContainerStarted","Data":"c011bbc4f42ff0859abb6d8adb8bae3e6e4c15f61f34d8289b1696a0956bdbec"} Jul 22 18:58:17.155653 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:17.155610 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55c7887b67-fph6t" podStartSLOduration=1.497294964 podStartE2EDuration="5.155596928s" totalImagesPullingTime="3.658301964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:58:17.154532142 +0000 UTC m=+100.066696211" watchObservedRunningTime="2026-07-22 18:58:17.155596928 +0000 UTC m=+100.067760997" Jul 22 18:58:17.176210 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:17.176163 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-59f988c449-c5x4p" podStartSLOduration=1.71281467 podStartE2EDuration="5.176152019s" totalImagesPullingTime="3.463337349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:58:17.175264749 +0000 UTC m=+100.087428817" watchObservedRunningTime="2026-07-22 18:58:17.176152019 +0000 UTC m=+100.088316087" Jul 22 18:58:18.943730 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:18.943692 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55c7887b67-fph6t"] Jul 22 18:58:18.980623 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:18.980587 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56f7ddd58d-z7t5s"] Jul 22 18:58:19.064967 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.064921 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f7ddd58d-z7t5s"] Jul 22 18:58:19.065113 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.065049 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.131319 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.131276 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-oauth-config\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.131319 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.131320 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-trusted-ca-bundle\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.131319 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.131339 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-oauth-serving-cert\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.131628 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.131406 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvk62\" (UniqueName: \"kubernetes.io/projected/9f8181b9-fd39-4d99-9c4a-5f645d884598-kube-api-access-qvk62\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.131628 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.131423 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-service-ca\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.131628 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.131438 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-serving-cert\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.131628 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.131477 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-config\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.232197 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.232105 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvk62\" (UniqueName: \"kubernetes.io/projected/9f8181b9-fd39-4d99-9c4a-5f645d884598-kube-api-access-qvk62\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.232197 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.232143 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-service-ca\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.232197 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.232163 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-serving-cert\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.232479 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.232225 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-config\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.232479 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.232365 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-oauth-config\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.232479 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.232396 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-trusted-ca-bundle\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.232479 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.232425 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-oauth-serving-cert\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.232982 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.232957 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-service-ca\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.233102 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.232961 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-config\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.233199 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.233178 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-oauth-serving-cert\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.233602 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.233580 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-trusted-ca-bundle\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.234861 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.234843 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-serving-cert\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.234942 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.234885 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-oauth-config\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.239922 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.239904 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvk62\" (UniqueName: \"kubernetes.io/projected/9f8181b9-fd39-4d99-9c4a-5f645d884598-kube-api-access-qvk62\") pod \"console-56f7ddd58d-z7t5s\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.374613 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.374568 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:19.547956 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:58:19.547921 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8181b9_fd39_4d99_9c4a_5f645d884598.slice/crio-4c120fc2f23681c82b0ab79169298f4fb6d7f71f7aa3746471c28011ec30d11d WatchSource:0}: Error finding container 4c120fc2f23681c82b0ab79169298f4fb6d7f71f7aa3746471c28011ec30d11d: Status 404 returned error can't find the container with id 4c120fc2f23681c82b0ab79169298f4fb6d7f71f7aa3746471c28011ec30d11d Jul 22 18:58:19.550152 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:19.550121 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f7ddd58d-z7t5s"] Jul 22 18:58:20.154134 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:20.154095 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f7ddd58d-z7t5s" event={"ID":"9f8181b9-fd39-4d99-9c4a-5f645d884598","Type":"ContainerStarted","Data":"ed294cdddc1eb73e039d2482a55b6e8bb1880efdc950be0f2f8be1e55565a85e"} Jul 22 18:58:20.154551 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:20.154185 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f7ddd58d-z7t5s" event={"ID":"9f8181b9-fd39-4d99-9c4a-5f645d884598","Type":"ContainerStarted","Data":"4c120fc2f23681c82b0ab79169298f4fb6d7f71f7aa3746471c28011ec30d11d"} Jul 22 18:58:20.172013 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:20.171560 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56f7ddd58d-z7t5s" podStartSLOduration=2.171540628 podStartE2EDuration="2.171540628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:58:20.170666536 +0000 UTC m=+103.082830617" watchObservedRunningTime="2026-07-22 18:58:20.171540628 +0000 UTC m=+103.083704700" Jul 22 18:58:22.875765 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:22.875725 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:29.375442 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:29.375405 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:29.375917 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:29.375479 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:29.380039 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:29.380014 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:30.202372 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:30.202343 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 18:58:35.219568 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:35.219534 2583 generic.go:356] "Generic (PLEG): container finished" podID="aa91b8fa-def9-4745-b1ec-66df9c2adc5a" containerID="14a0ac7986dfeb13070f375b8d64a29d1f7b4cf9d27c931ff2d74f872fa1017d" exitCode=0 Jul 22 18:58:35.219993 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:35.219591 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn" event={"ID":"aa91b8fa-def9-4745-b1ec-66df9c2adc5a","Type":"ContainerDied","Data":"14a0ac7986dfeb13070f375b8d64a29d1f7b4cf9d27c931ff2d74f872fa1017d"} Jul 22 18:58:35.219993 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:35.219869 2583 scope.go:117] "RemoveContainer" containerID="14a0ac7986dfeb13070f375b8d64a29d1f7b4cf9d27c931ff2d74f872fa1017d" Jul 22 18:58:36.225233 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:36.225184 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-jhcsn" event={"ID":"aa91b8fa-def9-4745-b1ec-66df9c2adc5a","Type":"ContainerStarted","Data":"25512a69ad65db920623b04d8ee8a9f58216fdccaf8bddec09009ae039929519"} Jul 22 18:58:41.506074 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.506000 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" podUID="e07a3932-ff41-4d65-b28d-885e260eec9a" containerName="registry" containerID="cri-o://570703a13cdd1166e4dad23d9e71ea61099a43c90c0d9b8c97f5822cd4b8a189" gracePeriod=30 Jul 22 18:58:41.757998 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.757930 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:58:41.848244 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.848208 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-bound-sa-token\") pod \"e07a3932-ff41-4d65-b28d-885e260eec9a\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " Jul 22 18:58:41.848433 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.848259 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e07a3932-ff41-4d65-b28d-885e260eec9a-ca-trust-extracted\") pod \"e07a3932-ff41-4d65-b28d-885e260eec9a\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " Jul 22 18:58:41.848433 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.848278 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f6gn\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-kube-api-access-6f6gn\") pod \"e07a3932-ff41-4d65-b28d-885e260eec9a\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " Jul 22 18:58:41.848433 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.848297 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-certificates\") pod \"e07a3932-ff41-4d65-b28d-885e260eec9a\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " Jul 22 18:58:41.848433 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.848322 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e07a3932-ff41-4d65-b28d-885e260eec9a-image-registry-private-configuration\") pod \"e07a3932-ff41-4d65-b28d-885e260eec9a\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " Jul 22 18:58:41.848433 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.848338 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e07a3932-ff41-4d65-b28d-885e260eec9a-trusted-ca\") pod \"e07a3932-ff41-4d65-b28d-885e260eec9a\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " Jul 22 18:58:41.848433 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.848366 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e07a3932-ff41-4d65-b28d-885e260eec9a-installation-pull-secrets\") pod \"e07a3932-ff41-4d65-b28d-885e260eec9a\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " Jul 22 18:58:41.848433 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.848404 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls\") pod \"e07a3932-ff41-4d65-b28d-885e260eec9a\" (UID: \"e07a3932-ff41-4d65-b28d-885e260eec9a\") " Jul 22 18:58:41.848864 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.848842 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e07a3932-ff41-4d65-b28d-885e260eec9a" (UID: "e07a3932-ff41-4d65-b28d-885e260eec9a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 18:58:41.849247 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.849158 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07a3932-ff41-4d65-b28d-885e260eec9a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e07a3932-ff41-4d65-b28d-885e260eec9a" (UID: "e07a3932-ff41-4d65-b28d-885e260eec9a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 18:58:41.851110 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.851081 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-kube-api-access-6f6gn" (OuterVolumeSpecName: "kube-api-access-6f6gn") pod "e07a3932-ff41-4d65-b28d-885e260eec9a" (UID: "e07a3932-ff41-4d65-b28d-885e260eec9a"). InnerVolumeSpecName "kube-api-access-6f6gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 18:58:41.851272 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.851240 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e07a3932-ff41-4d65-b28d-885e260eec9a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e07a3932-ff41-4d65-b28d-885e260eec9a" (UID: "e07a3932-ff41-4d65-b28d-885e260eec9a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 18:58:41.851379 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.851295 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e07a3932-ff41-4d65-b28d-885e260eec9a" (UID: "e07a3932-ff41-4d65-b28d-885e260eec9a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 18:58:41.851474 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.851454 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e07a3932-ff41-4d65-b28d-885e260eec9a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e07a3932-ff41-4d65-b28d-885e260eec9a" (UID: "e07a3932-ff41-4d65-b28d-885e260eec9a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 18:58:41.851560 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.851509 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e07a3932-ff41-4d65-b28d-885e260eec9a" (UID: "e07a3932-ff41-4d65-b28d-885e260eec9a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 18:58:41.856399 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.856355 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07a3932-ff41-4d65-b28d-885e260eec9a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e07a3932-ff41-4d65-b28d-885e260eec9a" (UID: "e07a3932-ff41-4d65-b28d-885e260eec9a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 18:58:41.949471 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.949431 2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-bound-sa-token\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:58:41.949471 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.949463 2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e07a3932-ff41-4d65-b28d-885e260eec9a-ca-trust-extracted\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:58:41.949471 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.949477 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f6gn\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-kube-api-access-6f6gn\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:58:41.949740 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.949489 2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-certificates\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:58:41.949740 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.949535 2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e07a3932-ff41-4d65-b28d-885e260eec9a-image-registry-private-configuration\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:58:41.949740 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.949547 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e07a3932-ff41-4d65-b28d-885e260eec9a-trusted-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:58:41.949740 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.949557 2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e07a3932-ff41-4d65-b28d-885e260eec9a-installation-pull-secrets\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:58:41.949740 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:41.949568 2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e07a3932-ff41-4d65-b28d-885e260eec9a-registry-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:58:42.249726 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:42.249689 2583 generic.go:356] "Generic (PLEG): container finished" podID="e07a3932-ff41-4d65-b28d-885e260eec9a" containerID="570703a13cdd1166e4dad23d9e71ea61099a43c90c0d9b8c97f5822cd4b8a189" exitCode=0 Jul 22 18:58:42.249891 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:42.249734 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" Jul 22 18:58:42.249891 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:42.249782 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" event={"ID":"e07a3932-ff41-4d65-b28d-885e260eec9a","Type":"ContainerDied","Data":"570703a13cdd1166e4dad23d9e71ea61099a43c90c0d9b8c97f5822cd4b8a189"} Jul 22 18:58:42.249891 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:42.249832 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c97f75d66-rdvdn" event={"ID":"e07a3932-ff41-4d65-b28d-885e260eec9a","Type":"ContainerDied","Data":"add5a24224b0b04645c0594b7565a099e81b18e847e335d337d82b57b2c89735"} Jul 22 18:58:42.249891 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:42.249854 2583 scope.go:117] "RemoveContainer" containerID="570703a13cdd1166e4dad23d9e71ea61099a43c90c0d9b8c97f5822cd4b8a189" Jul 22 18:58:42.259039 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:42.259020 2583 scope.go:117] "RemoveContainer" containerID="570703a13cdd1166e4dad23d9e71ea61099a43c90c0d9b8c97f5822cd4b8a189" Jul 22 18:58:42.259322 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:58:42.259300 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"570703a13cdd1166e4dad23d9e71ea61099a43c90c0d9b8c97f5822cd4b8a189\": container with ID starting with 570703a13cdd1166e4dad23d9e71ea61099a43c90c0d9b8c97f5822cd4b8a189 not found: ID does not exist" containerID="570703a13cdd1166e4dad23d9e71ea61099a43c90c0d9b8c97f5822cd4b8a189" Jul 22 18:58:42.259371 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:42.259332 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570703a13cdd1166e4dad23d9e71ea61099a43c90c0d9b8c97f5822cd4b8a189"} err="failed to get container status \"570703a13cdd1166e4dad23d9e71ea61099a43c90c0d9b8c97f5822cd4b8a189\": rpc error: code = NotFound desc = could not find container \"570703a13cdd1166e4dad23d9e71ea61099a43c90c0d9b8c97f5822cd4b8a189\": container with ID starting with 570703a13cdd1166e4dad23d9e71ea61099a43c90c0d9b8c97f5822cd4b8a189 not found: ID does not exist" Jul 22 18:58:42.271058 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:42.271032 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c97f75d66-rdvdn"] Jul 22 18:58:42.275222 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:42.275192 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-c97f75d66-rdvdn"] Jul 22 18:58:43.553947 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:43.553909 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e07a3932-ff41-4d65-b28d-885e260eec9a" path="/var/lib/kubelet/pods/e07a3932-ff41-4d65-b28d-885e260eec9a/volumes" Jul 22 18:58:44.168759 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.168644 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55c7887b67-fph6t" podUID="0334e485-8398-417e-834e-0def6853ae0d" containerName="console" containerID="cri-o://ecb0268cb4d2322f8e16a9b06500d152df6ebc33266f94689287190fe45f0614" gracePeriod=15 Jul 22 18:58:44.260138 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.260109 2583 generic.go:356] "Generic (PLEG): container finished" podID="1d766722-198a-46e6-9175-5c13a9138778" containerID="d30a775c542c665058f5436251433948631b2f9887f8d996ffd211025a618c27" exitCode=0 Jul 22 18:58:44.260319 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.260185 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc" event={"ID":"1d766722-198a-46e6-9175-5c13a9138778","Type":"ContainerDied","Data":"d30a775c542c665058f5436251433948631b2f9887f8d996ffd211025a618c27"} Jul 22 18:58:44.260600 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.260582 2583 scope.go:117] "RemoveContainer" containerID="d30a775c542c665058f5436251433948631b2f9887f8d996ffd211025a618c27" Jul 22 18:58:44.261772 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.261749 2583 generic.go:356] "Generic (PLEG): container finished" podID="e0aa0a66-a4d6-4557-bf05-05c34c80c173" containerID="c9f2cd033a80146753be47c360dc24a95f9eb905e4ed8c5c1747720a7b458313" exitCode=0 Jul 22 18:58:44.261862 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.261804 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-2nd4q" event={"ID":"e0aa0a66-a4d6-4557-bf05-05c34c80c173","Type":"ContainerDied","Data":"c9f2cd033a80146753be47c360dc24a95f9eb905e4ed8c5c1747720a7b458313"} Jul 22 18:58:44.262110 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.262094 2583 scope.go:117] "RemoveContainer" containerID="c9f2cd033a80146753be47c360dc24a95f9eb905e4ed8c5c1747720a7b458313" Jul 22 18:58:44.427357 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.427326 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:44.573085 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.573049 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-oauth-serving-cert\") pod \"0334e485-8398-417e-834e-0def6853ae0d\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " Jul 22 18:58:44.573566 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.573098 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-service-ca\") pod \"0334e485-8398-417e-834e-0def6853ae0d\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " Jul 22 18:58:44.573566 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.573159 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sg2n\" (UniqueName: \"kubernetes.io/projected/0334e485-8398-417e-834e-0def6853ae0d-kube-api-access-7sg2n\") pod \"0334e485-8398-417e-834e-0def6853ae0d\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " Jul 22 18:58:44.573566 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.573202 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0334e485-8398-417e-834e-0def6853ae0d-console-serving-cert\") pod \"0334e485-8398-417e-834e-0def6853ae0d\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " Jul 22 18:58:44.573566 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.573251 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-trusted-ca-bundle\") pod \"0334e485-8398-417e-834e-0def6853ae0d\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " Jul 22 18:58:44.573566 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.573283 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0334e485-8398-417e-834e-0def6853ae0d-console-oauth-config\") pod \"0334e485-8398-417e-834e-0def6853ae0d\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " Jul 22 18:58:44.573566 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.573339 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-console-config\") pod \"0334e485-8398-417e-834e-0def6853ae0d\" (UID: \"0334e485-8398-417e-834e-0def6853ae0d\") " Jul 22 18:58:44.573566 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.573447 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0334e485-8398-417e-834e-0def6853ae0d" (UID: "0334e485-8398-417e-834e-0def6853ae0d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 18:58:44.573912 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.573614 2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-oauth-serving-cert\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:58:44.573912 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.573595 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-service-ca" (OuterVolumeSpecName: "service-ca") pod "0334e485-8398-417e-834e-0def6853ae0d" (UID: "0334e485-8398-417e-834e-0def6853ae0d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 18:58:44.573912 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.573846 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-console-config" (OuterVolumeSpecName: "console-config") pod "0334e485-8398-417e-834e-0def6853ae0d" (UID: "0334e485-8398-417e-834e-0def6853ae0d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 18:58:44.574063 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.573983 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0334e485-8398-417e-834e-0def6853ae0d" (UID: "0334e485-8398-417e-834e-0def6853ae0d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 18:58:44.576000 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.575969 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0334e485-8398-417e-834e-0def6853ae0d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0334e485-8398-417e-834e-0def6853ae0d" (UID: "0334e485-8398-417e-834e-0def6853ae0d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 18:58:44.576331 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.576309 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0334e485-8398-417e-834e-0def6853ae0d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0334e485-8398-417e-834e-0def6853ae0d" (UID: "0334e485-8398-417e-834e-0def6853ae0d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 18:58:44.576458 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.576436 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0334e485-8398-417e-834e-0def6853ae0d-kube-api-access-7sg2n" (OuterVolumeSpecName: "kube-api-access-7sg2n") pod "0334e485-8398-417e-834e-0def6853ae0d" (UID: "0334e485-8398-417e-834e-0def6853ae0d"). InnerVolumeSpecName "kube-api-access-7sg2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 18:58:44.674932 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.674841 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-trusted-ca-bundle\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:58:44.674932 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.674877 2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0334e485-8398-417e-834e-0def6853ae0d-console-oauth-config\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:58:44.674932 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.674892 2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-console-config\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:58:44.674932 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.674906 2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0334e485-8398-417e-834e-0def6853ae0d-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:58:44.674932 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.674919 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sg2n\" (UniqueName: \"kubernetes.io/projected/0334e485-8398-417e-834e-0def6853ae0d-kube-api-access-7sg2n\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:58:44.674932 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:44.674932 2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0334e485-8398-417e-834e-0def6853ae0d-console-serving-cert\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:58:45.268563 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:45.268518 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-5rgfc" event={"ID":"1d766722-198a-46e6-9175-5c13a9138778","Type":"ContainerStarted","Data":"0dbb31607cdad23658d1cb616166bc0afb4845bcf31aa0d4792fe988c385a615"} Jul 22 18:58:45.270568 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:45.270541 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-2nd4q" event={"ID":"e0aa0a66-a4d6-4557-bf05-05c34c80c173","Type":"ContainerStarted","Data":"70b10d61cab61888a8b5aff62302413318ecb0d619905aa21cb9491b8c3bfa85"} Jul 22 18:58:45.271775 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:45.271747 2583 generic.go:356] "Generic (PLEG): container finished" podID="0334e485-8398-417e-834e-0def6853ae0d" containerID="ecb0268cb4d2322f8e16a9b06500d152df6ebc33266f94689287190fe45f0614" exitCode=0 Jul 22 18:58:45.271899 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:45.271792 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55c7887b67-fph6t" Jul 22 18:58:45.271899 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:45.271817 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55c7887b67-fph6t" event={"ID":"0334e485-8398-417e-834e-0def6853ae0d","Type":"ContainerDied","Data":"ecb0268cb4d2322f8e16a9b06500d152df6ebc33266f94689287190fe45f0614"} Jul 22 18:58:45.271899 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:45.271859 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55c7887b67-fph6t" event={"ID":"0334e485-8398-417e-834e-0def6853ae0d","Type":"ContainerDied","Data":"2402816420fa9f3e835a875084e73d09f869704b7b7a45f5c7871950372b4b19"} Jul 22 18:58:45.271899 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:45.271888 2583 scope.go:117] "RemoveContainer" containerID="ecb0268cb4d2322f8e16a9b06500d152df6ebc33266f94689287190fe45f0614" Jul 22 18:58:45.282975 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:45.282956 2583 scope.go:117] "RemoveContainer" containerID="ecb0268cb4d2322f8e16a9b06500d152df6ebc33266f94689287190fe45f0614" Jul 22 18:58:45.283268 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:58:45.283247 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb0268cb4d2322f8e16a9b06500d152df6ebc33266f94689287190fe45f0614\": container with ID starting with ecb0268cb4d2322f8e16a9b06500d152df6ebc33266f94689287190fe45f0614 not found: ID does not exist" containerID="ecb0268cb4d2322f8e16a9b06500d152df6ebc33266f94689287190fe45f0614" Jul 22 18:58:45.283357 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:45.283276 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb0268cb4d2322f8e16a9b06500d152df6ebc33266f94689287190fe45f0614"} err="failed to get container status \"ecb0268cb4d2322f8e16a9b06500d152df6ebc33266f94689287190fe45f0614\": rpc error: code = NotFound desc = could not find container \"ecb0268cb4d2322f8e16a9b06500d152df6ebc33266f94689287190fe45f0614\": container with ID starting with ecb0268cb4d2322f8e16a9b06500d152df6ebc33266f94689287190fe45f0614 not found: ID does not exist" Jul 22 18:58:45.318245 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:45.318208 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55c7887b67-fph6t"] Jul 22 18:58:45.322235 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:45.322201 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55c7887b67-fph6t"] Jul 22 18:58:45.549171 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:58:45.549079 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0334e485-8398-417e-834e-0def6853ae0d" path="/var/lib/kubelet/pods/0334e485-8398-417e-834e-0def6853ae0d/volumes" Jul 22 18:59:27.912761 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:27.912677 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 18:59:27.913551 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:27.912974 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="alertmanager" containerID="cri-o://ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b" gracePeriod=120 Jul 22 18:59:27.913551 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:27.913040 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="kube-rbac-proxy-metric" containerID="cri-o://07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a" gracePeriod=120 Jul 22 18:59:27.913551 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:27.913081 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="prom-label-proxy" containerID="cri-o://e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba" gracePeriod=120 Jul 22 18:59:27.913551 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:27.913117 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="config-reloader" containerID="cri-o://b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326" gracePeriod=120 Jul 22 18:59:27.913551 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:27.913040 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="kube-rbac-proxy-web" containerID="cri-o://c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd" gracePeriod=120 Jul 22 18:59:27.913551 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:27.913162 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="kube-rbac-proxy" containerID="cri-o://79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667" gracePeriod=120 Jul 22 18:59:27.966290 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:59:27.966230 2583 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a1a2ca_a41a_4c23_b65e_30264abdf11d.slice/crio-b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a1a2ca_a41a_4c23_b65e_30264abdf11d.slice/crio-conmon-b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a1a2ca_a41a_4c23_b65e_30264abdf11d.slice/crio-conmon-79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a1a2ca_a41a_4c23_b65e_30264abdf11d.slice/crio-conmon-ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a1a2ca_a41a_4c23_b65e_30264abdf11d.slice/crio-ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b.scope\": RecentStats: unable to find data in memory cache]" Jul 22 18:59:28.446775 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:28.446739 2583 generic.go:356] "Generic (PLEG): container finished" podID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerID="e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba" exitCode=0 Jul 22 18:59:28.446775 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:28.446762 2583 generic.go:356] "Generic (PLEG): container finished" podID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerID="79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667" exitCode=0 Jul 22 18:59:28.446775 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:28.446768 2583 generic.go:356] "Generic (PLEG): container finished" podID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerID="b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326" exitCode=0 Jul 22 18:59:28.446775 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:28.446774 2583 generic.go:356] "Generic (PLEG): container finished" podID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerID="ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b" exitCode=0 Jul 22 18:59:28.447026 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:28.446798 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48a1a2ca-a41a-4c23-b65e-30264abdf11d","Type":"ContainerDied","Data":"e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba"} Jul 22 18:59:28.447026 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:28.446871 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48a1a2ca-a41a-4c23-b65e-30264abdf11d","Type":"ContainerDied","Data":"79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667"} Jul 22 18:59:28.447026 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:28.446898 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48a1a2ca-a41a-4c23-b65e-30264abdf11d","Type":"ContainerDied","Data":"b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326"} Jul 22 18:59:28.447026 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:28.446921 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48a1a2ca-a41a-4c23-b65e-30264abdf11d","Type":"ContainerDied","Data":"ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b"} Jul 22 18:59:29.262972 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.262945 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.348453 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.348422 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-config-volume\") pod \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " Jul 22 18:59:29.348653 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.348595 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " Jul 22 18:59:29.348653 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.348631 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/48a1a2ca-a41a-4c23-b65e-30264abdf11d-alertmanager-main-db\") pod \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " Jul 22 18:59:29.348771 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.348660 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48a1a2ca-a41a-4c23-b65e-30264abdf11d-alertmanager-trusted-ca-bundle\") pod \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " Jul 22 18:59:29.348771 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.348699 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/48a1a2ca-a41a-4c23-b65e-30264abdf11d-tls-assets\") pod \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " Jul 22 18:59:29.348771 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.348725 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7pt8\" (UniqueName: \"kubernetes.io/projected/48a1a2ca-a41a-4c23-b65e-30264abdf11d-kube-api-access-w7pt8\") pod \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " Jul 22 18:59:29.348771 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.348758 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy-web\") pod \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " Jul 22 18:59:29.348958 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.348855 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/48a1a2ca-a41a-4c23-b65e-30264abdf11d-config-out\") pod \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " Jul 22 18:59:29.348958 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.348890 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy\") pod \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " Jul 22 18:59:29.348958 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.348918 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-cluster-tls-config\") pod \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " Jul 22 18:59:29.348958 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.348950 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-main-tls\") pod \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " Jul 22 18:59:29.349151 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.348952 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a1a2ca-a41a-4c23-b65e-30264abdf11d-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "48a1a2ca-a41a-4c23-b65e-30264abdf11d" (UID: "48a1a2ca-a41a-4c23-b65e-30264abdf11d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 18:59:29.349151 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.349002 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48a1a2ca-a41a-4c23-b65e-30264abdf11d-metrics-client-ca\") pod \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " Jul 22 18:59:29.349151 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.349036 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a1a2ca-a41a-4c23-b65e-30264abdf11d-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "48a1a2ca-a41a-4c23-b65e-30264abdf11d" (UID: "48a1a2ca-a41a-4c23-b65e-30264abdf11d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 18:59:29.349151 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.349093 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-web-config\") pod \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\" (UID: \"48a1a2ca-a41a-4c23-b65e-30264abdf11d\") " Jul 22 18:59:29.349676 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.349409 2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/48a1a2ca-a41a-4c23-b65e-30264abdf11d-alertmanager-main-db\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:59:29.349676 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.349433 2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48a1a2ca-a41a-4c23-b65e-30264abdf11d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:59:29.349851 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.349744 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a1a2ca-a41a-4c23-b65e-30264abdf11d-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "48a1a2ca-a41a-4c23-b65e-30264abdf11d" (UID: "48a1a2ca-a41a-4c23-b65e-30264abdf11d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 18:59:29.351724 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.351696 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "48a1a2ca-a41a-4c23-b65e-30264abdf11d" (UID: "48a1a2ca-a41a-4c23-b65e-30264abdf11d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 18:59:29.351850 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.351727 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "48a1a2ca-a41a-4c23-b65e-30264abdf11d" (UID: "48a1a2ca-a41a-4c23-b65e-30264abdf11d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 18:59:29.351996 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.351960 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "48a1a2ca-a41a-4c23-b65e-30264abdf11d" (UID: "48a1a2ca-a41a-4c23-b65e-30264abdf11d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 18:59:29.352533 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.352478 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-config-volume" (OuterVolumeSpecName: "config-volume") pod "48a1a2ca-a41a-4c23-b65e-30264abdf11d" (UID: "48a1a2ca-a41a-4c23-b65e-30264abdf11d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 18:59:29.352615 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.352549 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a1a2ca-a41a-4c23-b65e-30264abdf11d-config-out" (OuterVolumeSpecName: "config-out") pod "48a1a2ca-a41a-4c23-b65e-30264abdf11d" (UID: "48a1a2ca-a41a-4c23-b65e-30264abdf11d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 18:59:29.353313 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.353288 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a1a2ca-a41a-4c23-b65e-30264abdf11d-kube-api-access-w7pt8" (OuterVolumeSpecName: "kube-api-access-w7pt8") pod "48a1a2ca-a41a-4c23-b65e-30264abdf11d" (UID: "48a1a2ca-a41a-4c23-b65e-30264abdf11d"). InnerVolumeSpecName "kube-api-access-w7pt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 18:59:29.353313 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.353295 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a1a2ca-a41a-4c23-b65e-30264abdf11d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "48a1a2ca-a41a-4c23-b65e-30264abdf11d" (UID: "48a1a2ca-a41a-4c23-b65e-30264abdf11d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 18:59:29.353416 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.353389 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "48a1a2ca-a41a-4c23-b65e-30264abdf11d" (UID: "48a1a2ca-a41a-4c23-b65e-30264abdf11d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 18:59:29.358316 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.358291 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "48a1a2ca-a41a-4c23-b65e-30264abdf11d" (UID: "48a1a2ca-a41a-4c23-b65e-30264abdf11d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 18:59:29.365400 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.365375 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-web-config" (OuterVolumeSpecName: "web-config") pod "48a1a2ca-a41a-4c23-b65e-30264abdf11d" (UID: "48a1a2ca-a41a-4c23-b65e-30264abdf11d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 18:59:29.450149 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.450118 2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/48a1a2ca-a41a-4c23-b65e-30264abdf11d-tls-assets\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:59:29.450149 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.450143 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7pt8\" (UniqueName: \"kubernetes.io/projected/48a1a2ca-a41a-4c23-b65e-30264abdf11d-kube-api-access-w7pt8\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:59:29.450149 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.450158 2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:59:29.450440 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.450167 2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/48a1a2ca-a41a-4c23-b65e-30264abdf11d-config-out\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:59:29.450440 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.450176 2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:59:29.450440 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.450184 2583 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-cluster-tls-config\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:59:29.450440 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.450194 2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-main-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:59:29.450440 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.450202 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48a1a2ca-a41a-4c23-b65e-30264abdf11d-metrics-client-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:59:29.450440 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.450210 2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-web-config\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:59:29.450440 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.450218 2583 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:59:29.450440 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.450226 2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/48a1a2ca-a41a-4c23-b65e-30264abdf11d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 18:59:29.454368 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.454340 2583 generic.go:356] "Generic (PLEG): container finished" podID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerID="07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a" exitCode=0 Jul 22 18:59:29.454368 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.454366 2583 generic.go:356] "Generic (PLEG): container finished" podID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerID="c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd" exitCode=0 Jul 22 18:59:29.454565 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.454397 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48a1a2ca-a41a-4c23-b65e-30264abdf11d","Type":"ContainerDied","Data":"07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a"} Jul 22 18:59:29.454565 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.454412 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.454565 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.454454 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48a1a2ca-a41a-4c23-b65e-30264abdf11d","Type":"ContainerDied","Data":"c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd"} Jul 22 18:59:29.454565 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.454482 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48a1a2ca-a41a-4c23-b65e-30264abdf11d","Type":"ContainerDied","Data":"3b1e12d80d7b03c0819822604e41467d6f6f296f2e00448484152e00e4dc1fb1"} Jul 22 18:59:29.454565 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.454532 2583 scope.go:117] "RemoveContainer" containerID="e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba" Jul 22 18:59:29.462681 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.462656 2583 scope.go:117] "RemoveContainer" containerID="07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a" Jul 22 18:59:29.470104 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.470086 2583 scope.go:117] "RemoveContainer" containerID="79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667" Jul 22 18:59:29.477448 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.477428 2583 scope.go:117] "RemoveContainer" containerID="c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd" Jul 22 18:59:29.479272 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.479249 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 18:59:29.484070 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.484047 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 18:59:29.485454 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.485436 2583 scope.go:117] "RemoveContainer" containerID="b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326" Jul 22 18:59:29.492627 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.492610 2583 scope.go:117] "RemoveContainer" containerID="ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b" Jul 22 18:59:29.499589 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.499546 2583 scope.go:117] "RemoveContainer" containerID="547e447da426e717c1712e7dd4d3a0734188ce793abc645f7c39aa1497d34ae9" Jul 22 18:59:29.506772 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.506756 2583 scope.go:117] "RemoveContainer" containerID="e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba" Jul 22 18:59:29.507036 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:59:29.507014 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba\": container with ID starting with e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba not found: ID does not exist" containerID="e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba" Jul 22 18:59:29.507078 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.507044 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba"} err="failed to get container status \"e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba\": rpc error: code = NotFound desc = could not find container \"e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba\": container with ID starting with e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba not found: ID does not exist" Jul 22 18:59:29.507078 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.507063 2583 scope.go:117] "RemoveContainer" containerID="07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a" Jul 22 18:59:29.507290 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:59:29.507271 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a\": container with ID starting with 07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a not found: ID does not exist" containerID="07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a" Jul 22 18:59:29.507330 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.507297 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a"} err="failed to get container status \"07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a\": rpc error: code = NotFound desc = could not find container \"07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a\": container with ID starting with 07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a not found: ID does not exist" Jul 22 18:59:29.507330 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.507313 2583 scope.go:117] "RemoveContainer" containerID="79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667" Jul 22 18:59:29.507545 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:59:29.507529 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667\": container with ID starting with 79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667 not found: ID does not exist" containerID="79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667" Jul 22 18:59:29.507599 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.507550 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667"} err="failed to get container status \"79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667\": rpc error: code = NotFound desc = could not find container \"79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667\": container with ID starting with 79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667 not found: ID does not exist" Jul 22 18:59:29.507599 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.507564 2583 scope.go:117] "RemoveContainer" containerID="c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd" Jul 22 18:59:29.507780 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:59:29.507765 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd\": container with ID starting with c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd not found: ID does not exist" containerID="c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd" Jul 22 18:59:29.507821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.507794 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd"} err="failed to get container status \"c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd\": rpc error: code = NotFound desc = could not find container \"c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd\": container with ID starting with c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd not found: ID does not exist" Jul 22 18:59:29.507821 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.507805 2583 scope.go:117] "RemoveContainer" containerID="b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326" Jul 22 18:59:29.507991 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:59:29.507976 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326\": container with ID starting with b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326 not found: ID does not exist" containerID="b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326" Jul 22 18:59:29.508033 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.507993 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326"} err="failed to get container status \"b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326\": rpc error: code = NotFound desc = could not find container \"b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326\": container with ID starting with b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326 not found: ID does not exist" Jul 22 18:59:29.508033 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.508004 2583 scope.go:117] "RemoveContainer" containerID="ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b" Jul 22 18:59:29.508209 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:59:29.508191 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b\": container with ID starting with ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b not found: ID does not exist" containerID="ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b" Jul 22 18:59:29.508248 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.508214 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b"} err="failed to get container status \"ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b\": rpc error: code = NotFound desc = could not find container \"ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b\": container with ID starting with ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b not found: ID does not exist" Jul 22 18:59:29.508248 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.508228 2583 scope.go:117] "RemoveContainer" containerID="547e447da426e717c1712e7dd4d3a0734188ce793abc645f7c39aa1497d34ae9" Jul 22 18:59:29.508426 ip-10-0-139-51 kubenswrapper[2583]: E0722 18:59:29.508412 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547e447da426e717c1712e7dd4d3a0734188ce793abc645f7c39aa1497d34ae9\": container with ID starting with 547e447da426e717c1712e7dd4d3a0734188ce793abc645f7c39aa1497d34ae9 not found: ID does not exist" containerID="547e447da426e717c1712e7dd4d3a0734188ce793abc645f7c39aa1497d34ae9" Jul 22 18:59:29.508465 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.508428 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547e447da426e717c1712e7dd4d3a0734188ce793abc645f7c39aa1497d34ae9"} err="failed to get container status \"547e447da426e717c1712e7dd4d3a0734188ce793abc645f7c39aa1497d34ae9\": rpc error: code = NotFound desc = could not find container \"547e447da426e717c1712e7dd4d3a0734188ce793abc645f7c39aa1497d34ae9\": container with ID starting with 547e447da426e717c1712e7dd4d3a0734188ce793abc645f7c39aa1497d34ae9 not found: ID does not exist" Jul 22 18:59:29.508465 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.508439 2583 scope.go:117] "RemoveContainer" containerID="e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba" Jul 22 18:59:29.508717 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.508700 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba"} err="failed to get container status \"e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba\": rpc error: code = NotFound desc = could not find container \"e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba\": container with ID starting with e7cbcccf58e3aa5f8aabf6ee70f7ba746d907bb6623e82da34485869fe8b36ba not found: ID does not exist" Jul 22 18:59:29.508762 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.508718 2583 scope.go:117] "RemoveContainer" containerID="07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a" Jul 22 18:59:29.508938 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.508918 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a"} err="failed to get container status \"07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a\": rpc error: code = NotFound desc = could not find container \"07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a\": container with ID starting with 07584afb8e2aa42e20edd31832baa75dbf3520c324c638caa79913409fa43b0a not found: ID does not exist" Jul 22 18:59:29.509006 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.508941 2583 scope.go:117] "RemoveContainer" containerID="79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667" Jul 22 18:59:29.509148 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.509131 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667"} err="failed to get container status \"79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667\": rpc error: code = NotFound desc = could not find container \"79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667\": container with ID starting with 79efa00f99ddbf1ccca294e9404f7d2daefb7c0b2433e9467f1fd3339f914667 not found: ID does not exist" Jul 22 18:59:29.509195 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.509149 2583 scope.go:117] "RemoveContainer" containerID="c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd" Jul 22 18:59:29.509349 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.509332 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd"} err="failed to get container status \"c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd\": rpc error: code = NotFound desc = could not find container \"c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd\": container with ID starting with c0efa2ac2c76d53d9092c6de6d2f95b50f91b5dc42c3303ec25b74f937c544fd not found: ID does not exist" Jul 22 18:59:29.509388 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.509349 2583 scope.go:117] "RemoveContainer" containerID="b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326" Jul 22 18:59:29.509573 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.509557 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326"} err="failed to get container status \"b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326\": rpc error: code = NotFound desc = could not find container \"b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326\": container with ID starting with b5072a83a2616e7eed15e8879a8c6d39d833ab827b10d70018399f14a15d3326 not found: ID does not exist" Jul 22 18:59:29.509617 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.509574 2583 scope.go:117] "RemoveContainer" containerID="ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b" Jul 22 18:59:29.509764 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.509745 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b"} err="failed to get container status \"ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b\": rpc error: code = NotFound desc = could not find container \"ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b\": container with ID starting with ddb98925cdec1fabee0b06c4d7cd6db456b9e73a81b872cb8e43bdaf2b72dd5b not found: ID does not exist" Jul 22 18:59:29.509803 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.509765 2583 scope.go:117] "RemoveContainer" containerID="547e447da426e717c1712e7dd4d3a0734188ce793abc645f7c39aa1497d34ae9" Jul 22 18:59:29.509985 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.509969 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547e447da426e717c1712e7dd4d3a0734188ce793abc645f7c39aa1497d34ae9"} err="failed to get container status \"547e447da426e717c1712e7dd4d3a0734188ce793abc645f7c39aa1497d34ae9\": rpc error: code = NotFound desc = could not find container \"547e447da426e717c1712e7dd4d3a0734188ce793abc645f7c39aa1497d34ae9\": container with ID starting with 547e447da426e717c1712e7dd4d3a0734188ce793abc645f7c39aa1497d34ae9 not found: ID does not exist" Jul 22 18:59:29.514268 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514249 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 18:59:29.514570 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514557 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="kube-rbac-proxy" Jul 22 18:59:29.514615 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514573 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="kube-rbac-proxy" Jul 22 18:59:29.514615 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514584 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="kube-rbac-proxy-web" Jul 22 18:59:29.514615 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514592 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="kube-rbac-proxy-web" Jul 22 18:59:29.514615 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514606 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="prom-label-proxy" Jul 22 18:59:29.514615 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514612 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="prom-label-proxy" Jul 22 18:59:29.514755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514622 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="init-config-reloader" Jul 22 18:59:29.514755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514627 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="init-config-reloader" Jul 22 18:59:29.514755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514635 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0334e485-8398-417e-834e-0def6853ae0d" containerName="console" Jul 22 18:59:29.514755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514640 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0334e485-8398-417e-834e-0def6853ae0d" containerName="console" Jul 22 18:59:29.514755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514649 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e07a3932-ff41-4d65-b28d-885e260eec9a" containerName="registry" Jul 22 18:59:29.514755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514654 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e07a3932-ff41-4d65-b28d-885e260eec9a" containerName="registry" Jul 22 18:59:29.514755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514663 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="alertmanager" Jul 22 18:59:29.514755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514668 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="alertmanager" Jul 22 18:59:29.514755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514677 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="kube-rbac-proxy-metric" Jul 22 18:59:29.514755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514685 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="kube-rbac-proxy-metric" Jul 22 18:59:29.514755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514694 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="config-reloader" Jul 22 18:59:29.514755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514699 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="config-reloader" Jul 22 18:59:29.514755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514747 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="alertmanager" Jul 22 18:59:29.514755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514757 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="kube-rbac-proxy" Jul 22 18:59:29.515146 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514768 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="kube-rbac-proxy-web" Jul 22 18:59:29.515146 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514774 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="kube-rbac-proxy-metric" Jul 22 18:59:29.515146 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514781 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e07a3932-ff41-4d65-b28d-885e260eec9a" containerName="registry" Jul 22 18:59:29.515146 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514786 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="prom-label-proxy" Jul 22 18:59:29.515146 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514792 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0334e485-8398-417e-834e-0def6853ae0d" containerName="console" Jul 22 18:59:29.515146 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.514796 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" containerName="config-reloader" Jul 22 18:59:29.524762 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.524740 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.527296 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.527270 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 18:59:29.527510 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.527468 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 18:59:29.527633 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.527612 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 18:59:29.527830 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.527801 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 18:59:29.527934 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.527810 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 18:59:29.527934 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.527877 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 18:59:29.527934 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.527911 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 18:59:29.527934 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.527922 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 18:59:29.528158 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.528063 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kh8sb\"" Jul 22 18:59:29.533390 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.533276 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 18:59:29.534026 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.534006 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 18:59:29.549468 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.549438 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48a1a2ca-a41a-4c23-b65e-30264abdf11d" path="/var/lib/kubelet/pods/48a1a2ca-a41a-4c23-b65e-30264abdf11d/volumes" Jul 22 18:59:29.651816 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.651775 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.651975 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.651829 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/afc0226a-ddaf-4667-b107-a027ee2b8d89-config-out\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.651975 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.651885 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.651975 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.651922 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.651975 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.651948 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afc0226a-ddaf-4667-b107-a027ee2b8d89-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.651975 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.651971 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.652244 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.652045 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-config-volume\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.652244 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.652073 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/afc0226a-ddaf-4667-b107-a027ee2b8d89-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.652244 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.652124 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvx2q\" (UniqueName: \"kubernetes.io/projected/afc0226a-ddaf-4667-b107-a027ee2b8d89-kube-api-access-fvx2q\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.652244 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.652187 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/afc0226a-ddaf-4667-b107-a027ee2b8d89-tls-assets\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.652244 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.652211 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.652244 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.652234 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afc0226a-ddaf-4667-b107-a027ee2b8d89-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.652452 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.652255 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-web-config\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.753128 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.753020 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.753128 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.753076 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afc0226a-ddaf-4667-b107-a027ee2b8d89-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.753128 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.753097 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.753128 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.753131 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-config-volume\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.753464 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.753148 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/afc0226a-ddaf-4667-b107-a027ee2b8d89-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.753464 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.753170 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvx2q\" (UniqueName: \"kubernetes.io/projected/afc0226a-ddaf-4667-b107-a027ee2b8d89-kube-api-access-fvx2q\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.753464 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.753200 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/afc0226a-ddaf-4667-b107-a027ee2b8d89-tls-assets\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.753464 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.753219 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.753464 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.753250 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afc0226a-ddaf-4667-b107-a027ee2b8d89-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.753464 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.753274 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-web-config\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.753464 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.753321 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.753464 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.753352 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/afc0226a-ddaf-4667-b107-a027ee2b8d89-config-out\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.753464 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.753375 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.754103 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.753914 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/afc0226a-ddaf-4667-b107-a027ee2b8d89-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.754321 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.754291 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afc0226a-ddaf-4667-b107-a027ee2b8d89-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.756295 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.756263 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/afc0226a-ddaf-4667-b107-a027ee2b8d89-tls-assets\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.756405 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.756302 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.756568 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.756545 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.756639 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.756609 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-config-volume\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.756700 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.756635 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.756754 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.756694 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.756971 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.756948 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.757143 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.757120 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afc0226a-ddaf-4667-b107-a027ee2b8d89-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.757216 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.757202 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/afc0226a-ddaf-4667-b107-a027ee2b8d89-web-config\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.758189 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.758174 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/afc0226a-ddaf-4667-b107-a027ee2b8d89-config-out\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.762758 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.762736 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvx2q\" (UniqueName: \"kubernetes.io/projected/afc0226a-ddaf-4667-b107-a027ee2b8d89-kube-api-access-fvx2q\") pod \"alertmanager-main-0\" (UID: \"afc0226a-ddaf-4667-b107-a027ee2b8d89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.836444 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.836412 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 18:59:29.981040 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:29.981015 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 18:59:29.982531 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:59:29.982464 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc0226a_ddaf_4667_b107_a027ee2b8d89.slice/crio-a977c253af3d4979b3ff23a24fec91f5d967ac7728feb72d80d1e7584f649ded WatchSource:0}: Error finding container a977c253af3d4979b3ff23a24fec91f5d967ac7728feb72d80d1e7584f649ded: Status 404 returned error can't find the container with id a977c253af3d4979b3ff23a24fec91f5d967ac7728feb72d80d1e7584f649ded Jul 22 18:59:30.460167 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:30.460078 2583 generic.go:356] "Generic (PLEG): container finished" podID="afc0226a-ddaf-4667-b107-a027ee2b8d89" containerID="ab56d249c11e8d5085b473cd254d8810dc1f1c987fbc8d161bde2b76f08af512" exitCode=0 Jul 22 18:59:30.460167 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:30.460156 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"afc0226a-ddaf-4667-b107-a027ee2b8d89","Type":"ContainerDied","Data":"ab56d249c11e8d5085b473cd254d8810dc1f1c987fbc8d161bde2b76f08af512"} Jul 22 18:59:30.460657 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:30.460226 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"afc0226a-ddaf-4667-b107-a027ee2b8d89","Type":"ContainerStarted","Data":"a977c253af3d4979b3ff23a24fec91f5d967ac7728feb72d80d1e7584f649ded"} Jul 22 18:59:31.468962 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:31.468919 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"afc0226a-ddaf-4667-b107-a027ee2b8d89","Type":"ContainerStarted","Data":"8cd12bfcdea499603a56f180f67a1713e6e39d209cf171547cc21b004ea07f8d"} Jul 22 18:59:31.469329 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:31.468985 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"afc0226a-ddaf-4667-b107-a027ee2b8d89","Type":"ContainerStarted","Data":"c5ffbfcab0e760f8a395b4713d95d19cc04a0137ef1f63ebcd7366ff2e216e39"} Jul 22 18:59:31.469329 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:31.469009 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"afc0226a-ddaf-4667-b107-a027ee2b8d89","Type":"ContainerStarted","Data":"65996ac4b75a639c452dab8e8f07952af83921b6eb5ac5346d3d688e038723af"} Jul 22 18:59:31.469329 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:31.469036 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"afc0226a-ddaf-4667-b107-a027ee2b8d89","Type":"ContainerStarted","Data":"50021b475f0f8af61c2e8035a6a059a9ba0ac817301b293c4273e37ae8b7e075"} Jul 22 18:59:31.469329 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:31.469060 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"afc0226a-ddaf-4667-b107-a027ee2b8d89","Type":"ContainerStarted","Data":"b74e000980e397a19884223a3a76559272dd3fe0706abe895d894abf75be8e00"} Jul 22 18:59:31.469329 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:31.469084 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"afc0226a-ddaf-4667-b107-a027ee2b8d89","Type":"ContainerStarted","Data":"1b5f0e8f97977c7efac3b2c33ce8132e014d24560d4f5579987a7561ff6a0923"} Jul 22 18:59:31.498104 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:31.498023 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.482231029 podStartE2EDuration="2.498006197s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.775168ms" isStatefulPod=false podCreationTimestamp="2026-07-22 18:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:59:31.496146125 +0000 UTC m=+174.408310194" watchObservedRunningTime="2026-07-22 18:59:31.498006197 +0000 UTC m=+174.410170286" Jul 22 18:59:43.841356 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:43.841317 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86c97f59d-x9c5s"] Jul 22 18:59:43.844663 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:43.844644 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:43.869214 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:43.869181 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86c97f59d-x9c5s"] Jul 22 18:59:43.969595 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:43.969553 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-oauth-serving-cert\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:43.969595 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:43.969600 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-service-ca\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:43.969826 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:43.969632 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81e25488-346a-45b5-8b9a-350144d69c05-console-serving-cert\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:43.969826 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:43.969735 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-trusted-ca-bundle\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:43.969826 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:43.969767 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94pc8\" (UniqueName: \"kubernetes.io/projected/81e25488-346a-45b5-8b9a-350144d69c05-kube-api-access-94pc8\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:43.969927 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:43.969832 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81e25488-346a-45b5-8b9a-350144d69c05-console-oauth-config\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:43.969927 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:43.969861 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-console-config\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:44.070871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.070827 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-oauth-serving-cert\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:44.070871 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.070871 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-service-ca\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:44.071027 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.070890 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81e25488-346a-45b5-8b9a-350144d69c05-console-serving-cert\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:44.071027 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.070934 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-trusted-ca-bundle\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:44.071027 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.070953 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94pc8\" (UniqueName: \"kubernetes.io/projected/81e25488-346a-45b5-8b9a-350144d69c05-kube-api-access-94pc8\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:44.071027 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.070991 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81e25488-346a-45b5-8b9a-350144d69c05-console-oauth-config\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:44.071027 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.071015 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-console-config\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:44.071659 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.071628 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-service-ca\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:44.071953 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.071716 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-oauth-serving-cert\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:44.071953 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.071811 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-console-config\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:44.071953 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.071856 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-trusted-ca-bundle\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:44.073710 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.073676 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81e25488-346a-45b5-8b9a-350144d69c05-console-oauth-config\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:44.073818 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.073711 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81e25488-346a-45b5-8b9a-350144d69c05-console-serving-cert\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:44.081040 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.081016 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94pc8\" (UniqueName: \"kubernetes.io/projected/81e25488-346a-45b5-8b9a-350144d69c05-kube-api-access-94pc8\") pod \"console-86c97f59d-x9c5s\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:44.155598 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.155509 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:44.289755 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.289721 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86c97f59d-x9c5s"] Jul 22 18:59:44.292812 ip-10-0-139-51 kubenswrapper[2583]: W0722 18:59:44.292783 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e25488_346a_45b5_8b9a_350144d69c05.slice/crio-01a06c423b6d00c555169ec3a7a59a7eb8520d3cbe52b03d1cd8bc127fdf96d1 WatchSource:0}: Error finding container 01a06c423b6d00c555169ec3a7a59a7eb8520d3cbe52b03d1cd8bc127fdf96d1: Status 404 returned error can't find the container with id 01a06c423b6d00c555169ec3a7a59a7eb8520d3cbe52b03d1cd8bc127fdf96d1 Jul 22 18:59:44.526675 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.526642 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86c97f59d-x9c5s" event={"ID":"81e25488-346a-45b5-8b9a-350144d69c05","Type":"ContainerStarted","Data":"c210e13d094777720d1474078be25b5ecf383bc50e521a210cd8e039d48dbe36"} Jul 22 18:59:44.526845 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.526696 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86c97f59d-x9c5s" event={"ID":"81e25488-346a-45b5-8b9a-350144d69c05","Type":"ContainerStarted","Data":"01a06c423b6d00c555169ec3a7a59a7eb8520d3cbe52b03d1cd8bc127fdf96d1"} Jul 22 18:59:44.548982 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:44.548931 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86c97f59d-x9c5s" podStartSLOduration=1.548912583 podStartE2EDuration="1.548912583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:59:44.547314952 +0000 UTC m=+187.459479034" watchObservedRunningTime="2026-07-22 18:59:44.548912583 +0000 UTC m=+187.461076651" Jul 22 18:59:54.156468 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:54.156428 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:54.156964 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:54.156555 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:54.161430 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:54.161407 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:54.571155 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:54.571126 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 18:59:54.628477 ip-10-0-139-51 kubenswrapper[2583]: I0722 18:59:54.628442 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56f7ddd58d-z7t5s"] Jul 22 19:00:19.663287 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:19.663219 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56f7ddd58d-z7t5s" podUID="9f8181b9-fd39-4d99-9c4a-5f645d884598" containerName="console" containerID="cri-o://ed294cdddc1eb73e039d2482a55b6e8bb1880efdc950be0f2f8be1e55565a85e" gracePeriod=15 Jul 22 19:00:19.914717 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:19.914655 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 19:00:19.987324 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:19.987287 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-config\") pod \"9f8181b9-fd39-4d99-9c4a-5f645d884598\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " Jul 22 19:00:19.987554 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:19.987352 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-service-ca\") pod \"9f8181b9-fd39-4d99-9c4a-5f645d884598\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " Jul 22 19:00:19.987554 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:19.987392 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-oauth-config\") pod \"9f8181b9-fd39-4d99-9c4a-5f645d884598\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " Jul 22 19:00:19.987554 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:19.987436 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-trusted-ca-bundle\") pod \"9f8181b9-fd39-4d99-9c4a-5f645d884598\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " Jul 22 19:00:19.987554 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:19.987461 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-serving-cert\") pod \"9f8181b9-fd39-4d99-9c4a-5f645d884598\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " Jul 22 19:00:19.987554 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:19.987547 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvk62\" (UniqueName: \"kubernetes.io/projected/9f8181b9-fd39-4d99-9c4a-5f645d884598-kube-api-access-qvk62\") pod \"9f8181b9-fd39-4d99-9c4a-5f645d884598\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " Jul 22 19:00:19.987812 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:19.987601 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-oauth-serving-cert\") pod \"9f8181b9-fd39-4d99-9c4a-5f645d884598\" (UID: \"9f8181b9-fd39-4d99-9c4a-5f645d884598\") " Jul 22 19:00:19.987865 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:19.987817 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-config" (OuterVolumeSpecName: "console-config") pod "9f8181b9-fd39-4d99-9c4a-5f645d884598" (UID: "9f8181b9-fd39-4d99-9c4a-5f645d884598"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:00:19.987920 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:19.987888 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-service-ca" (OuterVolumeSpecName: "service-ca") pod "9f8181b9-fd39-4d99-9c4a-5f645d884598" (UID: "9f8181b9-fd39-4d99-9c4a-5f645d884598"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:00:19.987977 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:19.987915 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9f8181b9-fd39-4d99-9c4a-5f645d884598" (UID: "9f8181b9-fd39-4d99-9c4a-5f645d884598"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:00:19.988167 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:19.988144 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9f8181b9-fd39-4d99-9c4a-5f645d884598" (UID: "9f8181b9-fd39-4d99-9c4a-5f645d884598"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:00:19.989886 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:19.989851 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9f8181b9-fd39-4d99-9c4a-5f645d884598" (UID: "9f8181b9-fd39-4d99-9c4a-5f645d884598"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:00:19.989981 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:19.989906 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9f8181b9-fd39-4d99-9c4a-5f645d884598" (UID: "9f8181b9-fd39-4d99-9c4a-5f645d884598"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:00:19.989981 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:19.989915 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8181b9-fd39-4d99-9c4a-5f645d884598-kube-api-access-qvk62" (OuterVolumeSpecName: "kube-api-access-qvk62") pod "9f8181b9-fd39-4d99-9c4a-5f645d884598" (UID: "9f8181b9-fd39-4d99-9c4a-5f645d884598"). InnerVolumeSpecName "kube-api-access-qvk62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:00:20.089168 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:20.089117 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-trusted-ca-bundle\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:00:20.089168 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:20.089162 2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-serving-cert\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:00:20.089168 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:20.089171 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvk62\" (UniqueName: \"kubernetes.io/projected/9f8181b9-fd39-4d99-9c4a-5f645d884598-kube-api-access-qvk62\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:00:20.089168 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:20.089183 2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-oauth-serving-cert\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:00:20.089448 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:20.089192 2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-config\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:00:20.089448 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:20.089201 2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f8181b9-fd39-4d99-9c4a-5f645d884598-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:00:20.089448 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:20.089208 2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f8181b9-fd39-4d99-9c4a-5f645d884598-console-oauth-config\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:00:20.683269 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:20.683232 2583 generic.go:356] "Generic (PLEG): container finished" podID="9f8181b9-fd39-4d99-9c4a-5f645d884598" containerID="ed294cdddc1eb73e039d2482a55b6e8bb1880efdc950be0f2f8be1e55565a85e" exitCode=0 Jul 22 19:00:20.683269 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:20.683273 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f7ddd58d-z7t5s" event={"ID":"9f8181b9-fd39-4d99-9c4a-5f645d884598","Type":"ContainerDied","Data":"ed294cdddc1eb73e039d2482a55b6e8bb1880efdc950be0f2f8be1e55565a85e"} Jul 22 19:00:20.683829 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:20.683307 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f7ddd58d-z7t5s" event={"ID":"9f8181b9-fd39-4d99-9c4a-5f645d884598","Type":"ContainerDied","Data":"4c120fc2f23681c82b0ab79169298f4fb6d7f71f7aa3746471c28011ec30d11d"} Jul 22 19:00:20.683829 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:20.683324 2583 scope.go:117] "RemoveContainer" containerID="ed294cdddc1eb73e039d2482a55b6e8bb1880efdc950be0f2f8be1e55565a85e" Jul 22 19:00:20.683829 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:20.683276 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f7ddd58d-z7t5s" Jul 22 19:00:20.692443 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:20.692421 2583 scope.go:117] "RemoveContainer" containerID="ed294cdddc1eb73e039d2482a55b6e8bb1880efdc950be0f2f8be1e55565a85e" Jul 22 19:00:20.692748 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:00:20.692718 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed294cdddc1eb73e039d2482a55b6e8bb1880efdc950be0f2f8be1e55565a85e\": container with ID starting with ed294cdddc1eb73e039d2482a55b6e8bb1880efdc950be0f2f8be1e55565a85e not found: ID does not exist" containerID="ed294cdddc1eb73e039d2482a55b6e8bb1880efdc950be0f2f8be1e55565a85e" Jul 22 19:00:20.692848 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:20.692757 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed294cdddc1eb73e039d2482a55b6e8bb1880efdc950be0f2f8be1e55565a85e"} err="failed to get container status \"ed294cdddc1eb73e039d2482a55b6e8bb1880efdc950be0f2f8be1e55565a85e\": rpc error: code = NotFound desc = could not find container \"ed294cdddc1eb73e039d2482a55b6e8bb1880efdc950be0f2f8be1e55565a85e\": container with ID starting with ed294cdddc1eb73e039d2482a55b6e8bb1880efdc950be0f2f8be1e55565a85e not found: ID does not exist" Jul 22 19:00:20.704018 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:20.703990 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56f7ddd58d-z7t5s"] Jul 22 19:00:20.707029 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:20.707006 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56f7ddd58d-z7t5s"] Jul 22 19:00:21.549423 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:21.549391 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f8181b9-fd39-4d99-9c4a-5f645d884598" path="/var/lib/kubelet/pods/9f8181b9-fd39-4d99-9c4a-5f645d884598/volumes" Jul 22 19:00:40.190393 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.190355 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4"] Jul 22 19:00:40.190825 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.190713 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f8181b9-fd39-4d99-9c4a-5f645d884598" containerName="console" Jul 22 19:00:40.190825 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.190725 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f8181b9-fd39-4d99-9c4a-5f645d884598" containerName="console" Jul 22 19:00:40.190825 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.190785 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f8181b9-fd39-4d99-9c4a-5f645d884598" containerName="console" Jul 22 19:00:40.193583 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.193562 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" Jul 22 19:00:40.196664 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.196637 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 19:00:40.196664 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.196662 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pzbtc\"" Jul 22 19:00:40.197763 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.197748 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 19:00:40.206410 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.206385 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4"] Jul 22 19:00:40.369069 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.369036 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4\" (UID: \"28ff882a-62b4-4f18-9ecb-9e71c1f2145c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" Jul 22 19:00:40.369270 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.369108 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4\" (UID: \"28ff882a-62b4-4f18-9ecb-9e71c1f2145c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" Jul 22 19:00:40.369270 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.369133 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flfgl\" (UniqueName: \"kubernetes.io/projected/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-kube-api-access-flfgl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4\" (UID: \"28ff882a-62b4-4f18-9ecb-9e71c1f2145c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" Jul 22 19:00:40.470441 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.470328 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4\" (UID: \"28ff882a-62b4-4f18-9ecb-9e71c1f2145c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" Jul 22 19:00:40.470441 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.470378 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flfgl\" (UniqueName: \"kubernetes.io/projected/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-kube-api-access-flfgl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4\" (UID: \"28ff882a-62b4-4f18-9ecb-9e71c1f2145c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" Jul 22 19:00:40.470441 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.470425 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4\" (UID: \"28ff882a-62b4-4f18-9ecb-9e71c1f2145c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" Jul 22 19:00:40.470843 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.470819 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4\" (UID: \"28ff882a-62b4-4f18-9ecb-9e71c1f2145c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" Jul 22 19:00:40.470907 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.470826 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4\" (UID: \"28ff882a-62b4-4f18-9ecb-9e71c1f2145c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" Jul 22 19:00:40.480071 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.480042 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flfgl\" (UniqueName: \"kubernetes.io/projected/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-kube-api-access-flfgl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4\" (UID: \"28ff882a-62b4-4f18-9ecb-9e71c1f2145c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" Jul 22 19:00:40.502770 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.502733 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" Jul 22 19:00:40.637628 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.637599 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4"] Jul 22 19:00:40.639761 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:00:40.639735 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ff882a_62b4_4f18_9ecb_9e71c1f2145c.slice/crio-5762396d1c3fabe24e8526ccd925b70558b73c919a1383770d06aaac852d901a WatchSource:0}: Error finding container 5762396d1c3fabe24e8526ccd925b70558b73c919a1383770d06aaac852d901a: Status 404 returned error can't find the container with id 5762396d1c3fabe24e8526ccd925b70558b73c919a1383770d06aaac852d901a Jul 22 19:00:40.763887 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:40.763804 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" event={"ID":"28ff882a-62b4-4f18-9ecb-9e71c1f2145c","Type":"ContainerStarted","Data":"5762396d1c3fabe24e8526ccd925b70558b73c919a1383770d06aaac852d901a"} Jul 22 19:00:48.801655 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:48.801618 2583 generic.go:356] "Generic (PLEG): container finished" podID="28ff882a-62b4-4f18-9ecb-9e71c1f2145c" containerID="ee1800f17cfb1b6998cc6d1f6960dd303a7ba4cc6cc36fd62115c362d8c10656" exitCode=0 Jul 22 19:00:48.802187 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:48.801716 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" event={"ID":"28ff882a-62b4-4f18-9ecb-9e71c1f2145c","Type":"ContainerDied","Data":"ee1800f17cfb1b6998cc6d1f6960dd303a7ba4cc6cc36fd62115c362d8c10656"} Jul 22 19:00:51.817423 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:51.817382 2583 generic.go:356] "Generic (PLEG): container finished" podID="28ff882a-62b4-4f18-9ecb-9e71c1f2145c" containerID="3584cd4d2f5bd08ac63d0994d5fd07542cc435523e1f06b3e516afbd6f472a5b" exitCode=0 Jul 22 19:00:51.817926 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:51.817466 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" event={"ID":"28ff882a-62b4-4f18-9ecb-9e71c1f2145c","Type":"ContainerDied","Data":"3584cd4d2f5bd08ac63d0994d5fd07542cc435523e1f06b3e516afbd6f472a5b"} Jul 22 19:00:58.852938 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:58.852898 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" event={"ID":"28ff882a-62b4-4f18-9ecb-9e71c1f2145c","Type":"ContainerStarted","Data":"fb0a47450bfbe8d3d1b00692c149fd999d912d2c08e80bb91ce9db15c63ddd29"} Jul 22 19:00:58.872441 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:58.872384 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" podStartSLOduration=2.092988256 podStartE2EDuration="18.872369024s" totalImagesPullingTime="16.779380768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:00:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:00:58.870462174 +0000 UTC m=+261.782626242" watchObservedRunningTime="2026-07-22 19:00:58.872369024 +0000 UTC m=+261.784533091" Jul 22 19:00:59.859108 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:59.859072 2583 generic.go:356] "Generic (PLEG): container finished" podID="28ff882a-62b4-4f18-9ecb-9e71c1f2145c" containerID="fb0a47450bfbe8d3d1b00692c149fd999d912d2c08e80bb91ce9db15c63ddd29" exitCode=0 Jul 22 19:00:59.859533 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:00:59.859159 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" event={"ID":"28ff882a-62b4-4f18-9ecb-9e71c1f2145c","Type":"ContainerDied","Data":"fb0a47450bfbe8d3d1b00692c149fd999d912d2c08e80bb91ce9db15c63ddd29"} Jul 22 19:01:00.997987 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:00.997961 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" Jul 22 19:01:01.057955 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:01.057907 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-util\") pod \"28ff882a-62b4-4f18-9ecb-9e71c1f2145c\" (UID: \"28ff882a-62b4-4f18-9ecb-9e71c1f2145c\") " Jul 22 19:01:01.058216 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:01.057978 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-bundle\") pod \"28ff882a-62b4-4f18-9ecb-9e71c1f2145c\" (UID: \"28ff882a-62b4-4f18-9ecb-9e71c1f2145c\") " Jul 22 19:01:01.058216 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:01.057997 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flfgl\" (UniqueName: \"kubernetes.io/projected/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-kube-api-access-flfgl\") pod \"28ff882a-62b4-4f18-9ecb-9e71c1f2145c\" (UID: \"28ff882a-62b4-4f18-9ecb-9e71c1f2145c\") " Jul 22 19:01:01.058575 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:01.058551 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-bundle" (OuterVolumeSpecName: "bundle") pod "28ff882a-62b4-4f18-9ecb-9e71c1f2145c" (UID: "28ff882a-62b4-4f18-9ecb-9e71c1f2145c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:01:01.060320 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:01.060294 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-kube-api-access-flfgl" (OuterVolumeSpecName: "kube-api-access-flfgl") pod "28ff882a-62b4-4f18-9ecb-9e71c1f2145c" (UID: "28ff882a-62b4-4f18-9ecb-9e71c1f2145c"). InnerVolumeSpecName "kube-api-access-flfgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:01:01.062513 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:01.062476 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-util" (OuterVolumeSpecName: "util") pod "28ff882a-62b4-4f18-9ecb-9e71c1f2145c" (UID: "28ff882a-62b4-4f18-9ecb-9e71c1f2145c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:01:01.159109 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:01.159015 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-util\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:01:01.159109 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:01.159050 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-bundle\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:01:01.159109 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:01.159063 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flfgl\" (UniqueName: \"kubernetes.io/projected/28ff882a-62b4-4f18-9ecb-9e71c1f2145c-kube-api-access-flfgl\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:01:01.870781 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:01.870680 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" Jul 22 19:01:01.870781 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:01.870693 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zt2w4" event={"ID":"28ff882a-62b4-4f18-9ecb-9e71c1f2145c","Type":"ContainerDied","Data":"5762396d1c3fabe24e8526ccd925b70558b73c919a1383770d06aaac852d901a"} Jul 22 19:01:01.870781 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:01.870736 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5762396d1c3fabe24e8526ccd925b70558b73c919a1383770d06aaac852d901a" Jul 22 19:01:08.315237 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.315197 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6dlgz"] Jul 22 19:01:08.315731 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.315550 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ff882a-62b4-4f18-9ecb-9e71c1f2145c" containerName="util" Jul 22 19:01:08.315731 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.315562 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ff882a-62b4-4f18-9ecb-9e71c1f2145c" containerName="util" Jul 22 19:01:08.315731 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.315571 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ff882a-62b4-4f18-9ecb-9e71c1f2145c" containerName="extract" Jul 22 19:01:08.315731 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.315577 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ff882a-62b4-4f18-9ecb-9e71c1f2145c" containerName="extract" Jul 22 19:01:08.315731 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.315593 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ff882a-62b4-4f18-9ecb-9e71c1f2145c" containerName="pull" Jul 22 19:01:08.315731 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.315598 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ff882a-62b4-4f18-9ecb-9e71c1f2145c" containerName="pull" Jul 22 19:01:08.315731 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.315650 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ff882a-62b4-4f18-9ecb-9e71c1f2145c" containerName="extract" Jul 22 19:01:08.319078 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.319059 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6dlgz" Jul 22 19:01:08.321482 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.321457 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 22 19:01:08.321633 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.321578 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 22 19:01:08.321713 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.321638 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-2h4ns\"" Jul 22 19:01:08.329638 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.329613 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6dlgz"] Jul 22 19:01:08.426245 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.426202 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d7aa60b-ae74-4294-8d12-cb6b849d0319-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6dlgz\" (UID: \"0d7aa60b-ae74-4294-8d12-cb6b849d0319\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6dlgz" Jul 22 19:01:08.426391 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.426277 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxfj6\" (UniqueName: \"kubernetes.io/projected/0d7aa60b-ae74-4294-8d12-cb6b849d0319-kube-api-access-pxfj6\") pod \"cert-manager-operator-controller-manager-55c444bf86-6dlgz\" (UID: \"0d7aa60b-ae74-4294-8d12-cb6b849d0319\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6dlgz" Jul 22 19:01:08.527700 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.527668 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d7aa60b-ae74-4294-8d12-cb6b849d0319-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6dlgz\" (UID: \"0d7aa60b-ae74-4294-8d12-cb6b849d0319\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6dlgz" Jul 22 19:01:08.527843 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.527716 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxfj6\" (UniqueName: \"kubernetes.io/projected/0d7aa60b-ae74-4294-8d12-cb6b849d0319-kube-api-access-pxfj6\") pod \"cert-manager-operator-controller-manager-55c444bf86-6dlgz\" (UID: \"0d7aa60b-ae74-4294-8d12-cb6b849d0319\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6dlgz" Jul 22 19:01:08.528068 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.528048 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d7aa60b-ae74-4294-8d12-cb6b849d0319-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6dlgz\" (UID: \"0d7aa60b-ae74-4294-8d12-cb6b849d0319\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6dlgz" Jul 22 19:01:08.537404 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.537374 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxfj6\" (UniqueName: \"kubernetes.io/projected/0d7aa60b-ae74-4294-8d12-cb6b849d0319-kube-api-access-pxfj6\") pod \"cert-manager-operator-controller-manager-55c444bf86-6dlgz\" (UID: \"0d7aa60b-ae74-4294-8d12-cb6b849d0319\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6dlgz" Jul 22 19:01:08.629075 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.628984 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6dlgz" Jul 22 19:01:08.768844 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.768817 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6dlgz"] Jul 22 19:01:08.771516 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:01:08.771462 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7aa60b_ae74_4294_8d12_cb6b849d0319.slice/crio-377f9139e5c6152c45ffd524d3d0dabff0deb5965dfca900a54b82394693ff7d WatchSource:0}: Error finding container 377f9139e5c6152c45ffd524d3d0dabff0deb5965dfca900a54b82394693ff7d: Status 404 returned error can't find the container with id 377f9139e5c6152c45ffd524d3d0dabff0deb5965dfca900a54b82394693ff7d Jul 22 19:01:08.901030 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:08.900939 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6dlgz" event={"ID":"0d7aa60b-ae74-4294-8d12-cb6b849d0319","Type":"ContainerStarted","Data":"377f9139e5c6152c45ffd524d3d0dabff0deb5965dfca900a54b82394693ff7d"} Jul 22 19:01:10.912093 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:10.912058 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6dlgz" event={"ID":"0d7aa60b-ae74-4294-8d12-cb6b849d0319","Type":"ContainerStarted","Data":"0bf93bf5775b5715b2fd28c1acbe7246fbb8571952f4bb33b7065c8c77ad1415"} Jul 22 19:01:10.937328 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:10.937269 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6dlgz" podStartSLOduration=0.924282654 podStartE2EDuration="2.937250512s" totalImagesPullingTime="2.012967858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:01:10.935254374 +0000 UTC m=+273.847418442" watchObservedRunningTime="2026-07-22 19:01:10.937250512 +0000 UTC m=+273.849414558" Jul 22 19:01:12.209723 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.209685 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8"] Jul 22 19:01:12.212061 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.212043 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" Jul 22 19:01:12.215540 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.215521 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 19:01:12.215648 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.215538 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 19:01:12.216464 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.216440 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pzbtc\"" Jul 22 19:01:12.229839 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.229811 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8"] Jul 22 19:01:12.360736 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.360697 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkc55\" (UniqueName: \"kubernetes.io/projected/7d8a488d-7137-4108-9367-3c551c87e754-kube-api-access-nkc55\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8\" (UID: \"7d8a488d-7137-4108-9367-3c551c87e754\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" Jul 22 19:01:12.360929 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.360768 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d8a488d-7137-4108-9367-3c551c87e754-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8\" (UID: \"7d8a488d-7137-4108-9367-3c551c87e754\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" Jul 22 19:01:12.360929 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.360796 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d8a488d-7137-4108-9367-3c551c87e754-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8\" (UID: \"7d8a488d-7137-4108-9367-3c551c87e754\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" Jul 22 19:01:12.462163 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.462062 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkc55\" (UniqueName: \"kubernetes.io/projected/7d8a488d-7137-4108-9367-3c551c87e754-kube-api-access-nkc55\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8\" (UID: \"7d8a488d-7137-4108-9367-3c551c87e754\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" Jul 22 19:01:12.462163 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.462132 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d8a488d-7137-4108-9367-3c551c87e754-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8\" (UID: \"7d8a488d-7137-4108-9367-3c551c87e754\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" Jul 22 19:01:12.462163 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.462159 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d8a488d-7137-4108-9367-3c551c87e754-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8\" (UID: \"7d8a488d-7137-4108-9367-3c551c87e754\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" Jul 22 19:01:12.462580 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.462557 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d8a488d-7137-4108-9367-3c551c87e754-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8\" (UID: \"7d8a488d-7137-4108-9367-3c551c87e754\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" Jul 22 19:01:12.463185 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.462615 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d8a488d-7137-4108-9367-3c551c87e754-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8\" (UID: \"7d8a488d-7137-4108-9367-3c551c87e754\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" Jul 22 19:01:12.471027 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.470999 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkc55\" (UniqueName: \"kubernetes.io/projected/7d8a488d-7137-4108-9367-3c551c87e754-kube-api-access-nkc55\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8\" (UID: \"7d8a488d-7137-4108-9367-3c551c87e754\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" Jul 22 19:01:12.522036 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.521994 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" Jul 22 19:01:12.664927 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.664898 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8"] Jul 22 19:01:12.667315 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:01:12.667278 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8a488d_7137_4108_9367_3c551c87e754.slice/crio-b01db4f815324a700b3458762b8f72b4ba99b3d6e0abe0e3843d61fc89dc03c9 WatchSource:0}: Error finding container b01db4f815324a700b3458762b8f72b4ba99b3d6e0abe0e3843d61fc89dc03c9: Status 404 returned error can't find the container with id b01db4f815324a700b3458762b8f72b4ba99b3d6e0abe0e3843d61fc89dc03c9 Jul 22 19:01:12.922904 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.922868 2583 generic.go:356] "Generic (PLEG): container finished" podID="7d8a488d-7137-4108-9367-3c551c87e754" containerID="82bf71cbed06766b4ec3bb6a41e8cc639ea642bb4474038a59a2b7b45dd49968" exitCode=0 Jul 22 19:01:12.923071 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.922944 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" event={"ID":"7d8a488d-7137-4108-9367-3c551c87e754","Type":"ContainerDied","Data":"82bf71cbed06766b4ec3bb6a41e8cc639ea642bb4474038a59a2b7b45dd49968"} Jul 22 19:01:12.923071 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:12.922980 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" event={"ID":"7d8a488d-7137-4108-9367-3c551c87e754","Type":"ContainerStarted","Data":"b01db4f815324a700b3458762b8f72b4ba99b3d6e0abe0e3843d61fc89dc03c9"} Jul 22 19:01:14.769433 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:14.769398 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fxrc2"] Jul 22 19:01:14.771624 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:14.771606 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fxrc2" Jul 22 19:01:14.774042 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:14.774015 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 19:01:14.774172 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:14.774077 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 19:01:14.774172 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:14.774134 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8xrqj\"" Jul 22 19:01:14.782552 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:14.782527 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fxrc2"] Jul 22 19:01:14.883132 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:14.883097 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4277ac86-eabe-4cd8-a57a-5bbfd3f31aa9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fxrc2\" (UID: \"4277ac86-eabe-4cd8-a57a-5bbfd3f31aa9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fxrc2" Jul 22 19:01:14.883306 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:14.883209 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zppdh\" (UniqueName: \"kubernetes.io/projected/4277ac86-eabe-4cd8-a57a-5bbfd3f31aa9-kube-api-access-zppdh\") pod \"cert-manager-cainjector-559cdd8c87-fxrc2\" (UID: \"4277ac86-eabe-4cd8-a57a-5bbfd3f31aa9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fxrc2" Jul 22 19:01:14.983973 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:14.983936 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4277ac86-eabe-4cd8-a57a-5bbfd3f31aa9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fxrc2\" (UID: \"4277ac86-eabe-4cd8-a57a-5bbfd3f31aa9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fxrc2" Jul 22 19:01:14.984176 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:14.983984 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zppdh\" (UniqueName: \"kubernetes.io/projected/4277ac86-eabe-4cd8-a57a-5bbfd3f31aa9-kube-api-access-zppdh\") pod \"cert-manager-cainjector-559cdd8c87-fxrc2\" (UID: \"4277ac86-eabe-4cd8-a57a-5bbfd3f31aa9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fxrc2" Jul 22 19:01:14.992292 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:14.992255 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zppdh\" (UniqueName: \"kubernetes.io/projected/4277ac86-eabe-4cd8-a57a-5bbfd3f31aa9-kube-api-access-zppdh\") pod \"cert-manager-cainjector-559cdd8c87-fxrc2\" (UID: \"4277ac86-eabe-4cd8-a57a-5bbfd3f31aa9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fxrc2" Jul 22 19:01:14.992610 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:14.992592 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4277ac86-eabe-4cd8-a57a-5bbfd3f31aa9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fxrc2\" (UID: \"4277ac86-eabe-4cd8-a57a-5bbfd3f31aa9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fxrc2" Jul 22 19:01:15.095364 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:15.095284 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fxrc2" Jul 22 19:01:15.903000 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:15.902964 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fxrc2"] Jul 22 19:01:15.906287 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:01:15.906250 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4277ac86_eabe_4cd8_a57a_5bbfd3f31aa9.slice/crio-4a4c47574828d36467451586dfb6f7ec0f41fbefa7f3ca575409eba107c14279 WatchSource:0}: Error finding container 4a4c47574828d36467451586dfb6f7ec0f41fbefa7f3ca575409eba107c14279: Status 404 returned error can't find the container with id 4a4c47574828d36467451586dfb6f7ec0f41fbefa7f3ca575409eba107c14279 Jul 22 19:01:15.946223 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:15.946176 2583 generic.go:356] "Generic (PLEG): container finished" podID="7d8a488d-7137-4108-9367-3c551c87e754" containerID="194a9654e80aa31924148cc742bce6ee53517d232ba137277b2c7ae215fe3e7b" exitCode=0 Jul 22 19:01:15.946406 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:15.946261 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" event={"ID":"7d8a488d-7137-4108-9367-3c551c87e754","Type":"ContainerDied","Data":"194a9654e80aa31924148cc742bce6ee53517d232ba137277b2c7ae215fe3e7b"} Jul 22 19:01:15.947522 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:15.947480 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fxrc2" event={"ID":"4277ac86-eabe-4cd8-a57a-5bbfd3f31aa9","Type":"ContainerStarted","Data":"4a4c47574828d36467451586dfb6f7ec0f41fbefa7f3ca575409eba107c14279"} Jul 22 19:01:16.957970 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:16.957935 2583 generic.go:356] "Generic (PLEG): container finished" podID="7d8a488d-7137-4108-9367-3c551c87e754" containerID="3bcea5f88196db157257db15d672f57bbd13bc774b136f94ac627c957843dfbe" exitCode=0 Jul 22 19:01:16.958431 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:16.958016 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" event={"ID":"7d8a488d-7137-4108-9367-3c551c87e754","Type":"ContainerDied","Data":"3bcea5f88196db157257db15d672f57bbd13bc774b136f94ac627c957843dfbe"} Jul 22 19:01:18.119850 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:18.119820 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" Jul 22 19:01:18.214509 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:18.214410 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkc55\" (UniqueName: \"kubernetes.io/projected/7d8a488d-7137-4108-9367-3c551c87e754-kube-api-access-nkc55\") pod \"7d8a488d-7137-4108-9367-3c551c87e754\" (UID: \"7d8a488d-7137-4108-9367-3c551c87e754\") " Jul 22 19:01:18.214666 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:18.214622 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d8a488d-7137-4108-9367-3c551c87e754-bundle\") pod \"7d8a488d-7137-4108-9367-3c551c87e754\" (UID: \"7d8a488d-7137-4108-9367-3c551c87e754\") " Jul 22 19:01:18.214666 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:18.214656 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d8a488d-7137-4108-9367-3c551c87e754-util\") pod \"7d8a488d-7137-4108-9367-3c551c87e754\" (UID: \"7d8a488d-7137-4108-9367-3c551c87e754\") " Jul 22 19:01:18.215091 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:18.215062 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d8a488d-7137-4108-9367-3c551c87e754-bundle" (OuterVolumeSpecName: "bundle") pod "7d8a488d-7137-4108-9367-3c551c87e754" (UID: "7d8a488d-7137-4108-9367-3c551c87e754"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:01:18.217051 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:18.217023 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8a488d-7137-4108-9367-3c551c87e754-kube-api-access-nkc55" (OuterVolumeSpecName: "kube-api-access-nkc55") pod "7d8a488d-7137-4108-9367-3c551c87e754" (UID: "7d8a488d-7137-4108-9367-3c551c87e754"). InnerVolumeSpecName "kube-api-access-nkc55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:01:18.220932 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:18.220896 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d8a488d-7137-4108-9367-3c551c87e754-util" (OuterVolumeSpecName: "util") pod "7d8a488d-7137-4108-9367-3c551c87e754" (UID: "7d8a488d-7137-4108-9367-3c551c87e754"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:01:18.315741 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:18.315696 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d8a488d-7137-4108-9367-3c551c87e754-util\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:01:18.315741 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:18.315738 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkc55\" (UniqueName: \"kubernetes.io/projected/7d8a488d-7137-4108-9367-3c551c87e754-kube-api-access-nkc55\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:01:18.315741 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:18.315750 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d8a488d-7137-4108-9367-3c551c87e754-bundle\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:01:18.970858 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:18.970822 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" Jul 22 19:01:18.970858 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:18.970845 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwqbg8" event={"ID":"7d8a488d-7137-4108-9367-3c551c87e754","Type":"ContainerDied","Data":"b01db4f815324a700b3458762b8f72b4ba99b3d6e0abe0e3843d61fc89dc03c9"} Jul 22 19:01:18.971148 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:18.970898 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b01db4f815324a700b3458762b8f72b4ba99b3d6e0abe0e3843d61fc89dc03c9" Jul 22 19:01:19.976585 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:19.976547 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fxrc2" event={"ID":"4277ac86-eabe-4cd8-a57a-5bbfd3f31aa9","Type":"ContainerStarted","Data":"f84314318f0c67d5f49a811376ad47d288a224b38770931a28bbf44b7c114814"} Jul 22 19:01:19.994835 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:19.994781 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fxrc2" podStartSLOduration=2.808568942 podStartE2EDuration="5.994766861s" totalImagesPullingTime="3.186197919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:01:19.993658939 +0000 UTC m=+282.905823018" watchObservedRunningTime="2026-07-22 19:01:19.994766861 +0000 UTC m=+282.906930928" Jul 22 19:01:25.306405 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.306367 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pcxq"] Jul 22 19:01:25.306826 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.306704 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d8a488d-7137-4108-9367-3c551c87e754" containerName="pull" Jul 22 19:01:25.306826 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.306715 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d8a488d-7137-4108-9367-3c551c87e754" containerName="pull" Jul 22 19:01:25.306826 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.306724 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d8a488d-7137-4108-9367-3c551c87e754" containerName="util" Jul 22 19:01:25.306826 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.306730 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d8a488d-7137-4108-9367-3c551c87e754" containerName="util" Jul 22 19:01:25.306826 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.306752 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d8a488d-7137-4108-9367-3c551c87e754" containerName="extract" Jul 22 19:01:25.306826 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.306757 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d8a488d-7137-4108-9367-3c551c87e754" containerName="extract" Jul 22 19:01:25.306826 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.306813 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d8a488d-7137-4108-9367-3c551c87e754" containerName="extract" Jul 22 19:01:25.310250 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.310232 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pcxq" Jul 22 19:01:25.314908 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.314889 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 19:01:25.315830 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.315812 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 19:01:25.316015 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.315995 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7bzrd\"" Jul 22 19:01:25.325763 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.325739 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pcxq"] Jul 22 19:01:25.387448 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.387404 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8cd6344-3e70-4fa9-8f70-49e5e8dc1ac2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6pcxq\" (UID: \"b8cd6344-3e70-4fa9-8f70-49e5e8dc1ac2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pcxq" Jul 22 19:01:25.387656 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.387462 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnrll\" (UniqueName: \"kubernetes.io/projected/b8cd6344-3e70-4fa9-8f70-49e5e8dc1ac2-kube-api-access-hnrll\") pod \"openshift-lws-operator-fd8ccff4c-6pcxq\" (UID: \"b8cd6344-3e70-4fa9-8f70-49e5e8dc1ac2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pcxq" Jul 22 19:01:25.488117 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.488077 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8cd6344-3e70-4fa9-8f70-49e5e8dc1ac2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6pcxq\" (UID: \"b8cd6344-3e70-4fa9-8f70-49e5e8dc1ac2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pcxq" Jul 22 19:01:25.488316 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.488136 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnrll\" (UniqueName: \"kubernetes.io/projected/b8cd6344-3e70-4fa9-8f70-49e5e8dc1ac2-kube-api-access-hnrll\") pod \"openshift-lws-operator-fd8ccff4c-6pcxq\" (UID: \"b8cd6344-3e70-4fa9-8f70-49e5e8dc1ac2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pcxq" Jul 22 19:01:25.488465 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.488444 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8cd6344-3e70-4fa9-8f70-49e5e8dc1ac2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6pcxq\" (UID: \"b8cd6344-3e70-4fa9-8f70-49e5e8dc1ac2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pcxq" Jul 22 19:01:25.501428 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.501396 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnrll\" (UniqueName: \"kubernetes.io/projected/b8cd6344-3e70-4fa9-8f70-49e5e8dc1ac2-kube-api-access-hnrll\") pod \"openshift-lws-operator-fd8ccff4c-6pcxq\" (UID: \"b8cd6344-3e70-4fa9-8f70-49e5e8dc1ac2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pcxq" Jul 22 19:01:25.619016 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.618915 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pcxq" Jul 22 19:01:25.761232 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:25.761206 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pcxq"] Jul 22 19:01:25.763739 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:01:25.763700 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8cd6344_3e70_4fa9_8f70_49e5e8dc1ac2.slice/crio-4c8928b836276a024bc7039ca5859c6be9fd6ad419fc958007ad93627ef4f43b WatchSource:0}: Error finding container 4c8928b836276a024bc7039ca5859c6be9fd6ad419fc958007ad93627ef4f43b: Status 404 returned error can't find the container with id 4c8928b836276a024bc7039ca5859c6be9fd6ad419fc958007ad93627ef4f43b Jul 22 19:01:26.005876 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:26.005786 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pcxq" event={"ID":"b8cd6344-3e70-4fa9-8f70-49e5e8dc1ac2","Type":"ContainerStarted","Data":"4c8928b836276a024bc7039ca5859c6be9fd6ad419fc958007ad93627ef4f43b"} Jul 22 19:01:31.035603 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:31.035562 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pcxq" event={"ID":"b8cd6344-3e70-4fa9-8f70-49e5e8dc1ac2","Type":"ContainerStarted","Data":"e884b9ab84ac734df802d370906603e5c6959171aea91d9ed096d2f5a2632362"} Jul 22 19:01:31.053310 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:31.053240 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pcxq" podStartSLOduration=1.0670665480000001 podStartE2EDuration="6.053218364s" totalImagesPullingTime="4.986151816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:01:31.051021697 +0000 UTC m=+293.963185766" watchObservedRunningTime="2026-07-22 19:01:31.053218364 +0000 UTC m=+293.965382433" Jul 22 19:01:33.096139 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.096103 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k"] Jul 22 19:01:33.099011 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.098993 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" Jul 22 19:01:33.101920 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.101892 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 19:01:33.101920 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.101893 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 19:01:33.102081 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.101937 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pzbtc\"" Jul 22 19:01:33.108683 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.108658 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k"] Jul 22 19:01:33.145452 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.145415 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51016d0d-f009-4fe3-97d0-88ead52c5f18-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k\" (UID: \"51016d0d-f009-4fe3-97d0-88ead52c5f18\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" Jul 22 19:01:33.145659 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.145477 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51016d0d-f009-4fe3-97d0-88ead52c5f18-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k\" (UID: \"51016d0d-f009-4fe3-97d0-88ead52c5f18\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" Jul 22 19:01:33.145659 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.145535 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tdtc\" (UniqueName: \"kubernetes.io/projected/51016d0d-f009-4fe3-97d0-88ead52c5f18-kube-api-access-8tdtc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k\" (UID: \"51016d0d-f009-4fe3-97d0-88ead52c5f18\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" Jul 22 19:01:33.246583 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.246544 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51016d0d-f009-4fe3-97d0-88ead52c5f18-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k\" (UID: \"51016d0d-f009-4fe3-97d0-88ead52c5f18\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" Jul 22 19:01:33.246776 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.246634 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51016d0d-f009-4fe3-97d0-88ead52c5f18-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k\" (UID: \"51016d0d-f009-4fe3-97d0-88ead52c5f18\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" Jul 22 19:01:33.246776 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.246661 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tdtc\" (UniqueName: \"kubernetes.io/projected/51016d0d-f009-4fe3-97d0-88ead52c5f18-kube-api-access-8tdtc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k\" (UID: \"51016d0d-f009-4fe3-97d0-88ead52c5f18\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" Jul 22 19:01:33.246956 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.246934 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51016d0d-f009-4fe3-97d0-88ead52c5f18-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k\" (UID: \"51016d0d-f009-4fe3-97d0-88ead52c5f18\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" Jul 22 19:01:33.247024 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.246983 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51016d0d-f009-4fe3-97d0-88ead52c5f18-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k\" (UID: \"51016d0d-f009-4fe3-97d0-88ead52c5f18\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" Jul 22 19:01:33.255692 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.255661 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tdtc\" (UniqueName: \"kubernetes.io/projected/51016d0d-f009-4fe3-97d0-88ead52c5f18-kube-api-access-8tdtc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k\" (UID: \"51016d0d-f009-4fe3-97d0-88ead52c5f18\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" Jul 22 19:01:33.408626 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.408534 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" Jul 22 19:01:33.549228 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:01:33.549195 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51016d0d_f009_4fe3_97d0_88ead52c5f18.slice/crio-85e8bb8c1660c34f83f74fbc1830234b881b089b7351f20364dd006485c31849 WatchSource:0}: Error finding container 85e8bb8c1660c34f83f74fbc1830234b881b089b7351f20364dd006485c31849: Status 404 returned error can't find the container with id 85e8bb8c1660c34f83f74fbc1830234b881b089b7351f20364dd006485c31849 Jul 22 19:01:33.551100 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:33.551067 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k"] Jul 22 19:01:34.050105 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:34.050071 2583 generic.go:356] "Generic (PLEG): container finished" podID="51016d0d-f009-4fe3-97d0-88ead52c5f18" containerID="b3f6440f42ff8e5bfe7e95c28a555c759403762c6c748ca661e67f47562c94af" exitCode=0 Jul 22 19:01:34.050285 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:34.050117 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" event={"ID":"51016d0d-f009-4fe3-97d0-88ead52c5f18","Type":"ContainerDied","Data":"b3f6440f42ff8e5bfe7e95c28a555c759403762c6c748ca661e67f47562c94af"} Jul 22 19:01:34.050285 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:34.050165 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" event={"ID":"51016d0d-f009-4fe3-97d0-88ead52c5f18","Type":"ContainerStarted","Data":"85e8bb8c1660c34f83f74fbc1830234b881b089b7351f20364dd006485c31849"} Jul 22 19:01:35.056697 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:35.056656 2583 generic.go:356] "Generic (PLEG): container finished" podID="51016d0d-f009-4fe3-97d0-88ead52c5f18" containerID="deb02511b1250b54088d64714f7f96e708162f3e8d491b548a6facb7590fa1c7" exitCode=0 Jul 22 19:01:35.057079 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:35.056738 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" event={"ID":"51016d0d-f009-4fe3-97d0-88ead52c5f18","Type":"ContainerDied","Data":"deb02511b1250b54088d64714f7f96e708162f3e8d491b548a6facb7590fa1c7"} Jul 22 19:01:36.063530 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:36.063474 2583 generic.go:356] "Generic (PLEG): container finished" podID="51016d0d-f009-4fe3-97d0-88ead52c5f18" containerID="e4e70bbfc877d0328c0e620010ad5a973a18745e191141ddad6009f42573e095" exitCode=0 Jul 22 19:01:36.063530 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:36.063533 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" event={"ID":"51016d0d-f009-4fe3-97d0-88ead52c5f18","Type":"ContainerDied","Data":"e4e70bbfc877d0328c0e620010ad5a973a18745e191141ddad6009f42573e095"} Jul 22 19:01:37.196528 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:37.196482 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" Jul 22 19:01:37.279467 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:37.279432 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tdtc\" (UniqueName: \"kubernetes.io/projected/51016d0d-f009-4fe3-97d0-88ead52c5f18-kube-api-access-8tdtc\") pod \"51016d0d-f009-4fe3-97d0-88ead52c5f18\" (UID: \"51016d0d-f009-4fe3-97d0-88ead52c5f18\") " Jul 22 19:01:37.279681 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:37.279482 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51016d0d-f009-4fe3-97d0-88ead52c5f18-bundle\") pod \"51016d0d-f009-4fe3-97d0-88ead52c5f18\" (UID: \"51016d0d-f009-4fe3-97d0-88ead52c5f18\") " Jul 22 19:01:37.279681 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:37.279532 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51016d0d-f009-4fe3-97d0-88ead52c5f18-util\") pod \"51016d0d-f009-4fe3-97d0-88ead52c5f18\" (UID: \"51016d0d-f009-4fe3-97d0-88ead52c5f18\") " Jul 22 19:01:37.280245 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:37.280216 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51016d0d-f009-4fe3-97d0-88ead52c5f18-bundle" (OuterVolumeSpecName: "bundle") pod "51016d0d-f009-4fe3-97d0-88ead52c5f18" (UID: "51016d0d-f009-4fe3-97d0-88ead52c5f18"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:01:37.281837 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:37.281803 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51016d0d-f009-4fe3-97d0-88ead52c5f18-kube-api-access-8tdtc" (OuterVolumeSpecName: "kube-api-access-8tdtc") pod "51016d0d-f009-4fe3-97d0-88ead52c5f18" (UID: "51016d0d-f009-4fe3-97d0-88ead52c5f18"). InnerVolumeSpecName "kube-api-access-8tdtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:01:37.285074 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:37.285030 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51016d0d-f009-4fe3-97d0-88ead52c5f18-util" (OuterVolumeSpecName: "util") pod "51016d0d-f009-4fe3-97d0-88ead52c5f18" (UID: "51016d0d-f009-4fe3-97d0-88ead52c5f18"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:01:37.380470 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:37.380383 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tdtc\" (UniqueName: \"kubernetes.io/projected/51016d0d-f009-4fe3-97d0-88ead52c5f18-kube-api-access-8tdtc\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:01:37.380470 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:37.380415 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51016d0d-f009-4fe3-97d0-88ead52c5f18-bundle\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:01:37.380470 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:37.380424 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51016d0d-f009-4fe3-97d0-88ead52c5f18-util\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:01:37.464224 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:37.464194 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:01:37.465731 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:37.465707 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:01:37.475511 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:37.475477 2583 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 19:01:38.074516 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:38.074458 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" Jul 22 19:01:38.074516 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:38.074469 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af64k2k" event={"ID":"51016d0d-f009-4fe3-97d0-88ead52c5f18","Type":"ContainerDied","Data":"85e8bb8c1660c34f83f74fbc1830234b881b089b7351f20364dd006485c31849"} Jul 22 19:01:38.074752 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:38.074574 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85e8bb8c1660c34f83f74fbc1830234b881b089b7351f20364dd006485c31849" Jul 22 19:01:50.774831 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.774795 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk"] Jul 22 19:01:50.775247 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.775154 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51016d0d-f009-4fe3-97d0-88ead52c5f18" containerName="pull" Jul 22 19:01:50.775247 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.775166 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="51016d0d-f009-4fe3-97d0-88ead52c5f18" containerName="pull" Jul 22 19:01:50.775247 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.775176 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51016d0d-f009-4fe3-97d0-88ead52c5f18" containerName="extract" Jul 22 19:01:50.775247 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.775181 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="51016d0d-f009-4fe3-97d0-88ead52c5f18" containerName="extract" Jul 22 19:01:50.775247 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.775189 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51016d0d-f009-4fe3-97d0-88ead52c5f18" containerName="util" Jul 22 19:01:50.775247 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.775196 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="51016d0d-f009-4fe3-97d0-88ead52c5f18" containerName="util" Jul 22 19:01:50.775440 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.775259 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="51016d0d-f009-4fe3-97d0-88ead52c5f18" containerName="extract" Jul 22 19:01:50.777272 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.777243 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:50.780757 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.780738 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 19:01:50.780885 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.780764 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 19:01:50.781972 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.781949 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lzcck\"" Jul 22 19:01:50.782087 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.782020 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 19:01:50.784347 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.784235 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 19:01:50.832515 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.832460 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk"] Jul 22 19:01:50.898506 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.898468 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fea8ac18-f5da-4d52-b696-c912e93c370a-tmp\") pod \"opendatahub-operator-controller-manager-cb586fcb5-k8xhk\" (UID: \"fea8ac18-f5da-4d52-b696-c912e93c370a\") " pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:50.898734 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.898554 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fea8ac18-f5da-4d52-b696-c912e93c370a-webhook-cert\") pod \"opendatahub-operator-controller-manager-cb586fcb5-k8xhk\" (UID: \"fea8ac18-f5da-4d52-b696-c912e93c370a\") " pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:50.898734 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.898614 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fea8ac18-f5da-4d52-b696-c912e93c370a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cb586fcb5-k8xhk\" (UID: \"fea8ac18-f5da-4d52-b696-c912e93c370a\") " pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:50.898734 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.898643 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fea8ac18-f5da-4d52-b696-c912e93c370a-manifests\") pod \"opendatahub-operator-controller-manager-cb586fcb5-k8xhk\" (UID: \"fea8ac18-f5da-4d52-b696-c912e93c370a\") " pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:50.898734 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.898668 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s75bl\" (UniqueName: \"kubernetes.io/projected/fea8ac18-f5da-4d52-b696-c912e93c370a-kube-api-access-s75bl\") pod \"opendatahub-operator-controller-manager-cb586fcb5-k8xhk\" (UID: \"fea8ac18-f5da-4d52-b696-c912e93c370a\") " pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:50.999914 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.999877 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fea8ac18-f5da-4d52-b696-c912e93c370a-tmp\") pod \"opendatahub-operator-controller-manager-cb586fcb5-k8xhk\" (UID: \"fea8ac18-f5da-4d52-b696-c912e93c370a\") " pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:50.999914 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:50.999909 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fea8ac18-f5da-4d52-b696-c912e93c370a-webhook-cert\") pod \"opendatahub-operator-controller-manager-cb586fcb5-k8xhk\" (UID: \"fea8ac18-f5da-4d52-b696-c912e93c370a\") " pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:51.000163 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:51.000058 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fea8ac18-f5da-4d52-b696-c912e93c370a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cb586fcb5-k8xhk\" (UID: \"fea8ac18-f5da-4d52-b696-c912e93c370a\") " pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:51.000163 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:51.000099 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fea8ac18-f5da-4d52-b696-c912e93c370a-manifests\") pod \"opendatahub-operator-controller-manager-cb586fcb5-k8xhk\" (UID: \"fea8ac18-f5da-4d52-b696-c912e93c370a\") " pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:51.000163 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:51.000127 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s75bl\" (UniqueName: \"kubernetes.io/projected/fea8ac18-f5da-4d52-b696-c912e93c370a-kube-api-access-s75bl\") pod \"opendatahub-operator-controller-manager-cb586fcb5-k8xhk\" (UID: \"fea8ac18-f5da-4d52-b696-c912e93c370a\") " pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:51.000481 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:51.000454 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fea8ac18-f5da-4d52-b696-c912e93c370a-manifests\") pod \"opendatahub-operator-controller-manager-cb586fcb5-k8xhk\" (UID: \"fea8ac18-f5da-4d52-b696-c912e93c370a\") " pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:51.002460 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:51.002433 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fea8ac18-f5da-4d52-b696-c912e93c370a-tmp\") pod \"opendatahub-operator-controller-manager-cb586fcb5-k8xhk\" (UID: \"fea8ac18-f5da-4d52-b696-c912e93c370a\") " pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:51.002658 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:51.002637 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fea8ac18-f5da-4d52-b696-c912e93c370a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cb586fcb5-k8xhk\" (UID: \"fea8ac18-f5da-4d52-b696-c912e93c370a\") " pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:51.002701 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:51.002679 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fea8ac18-f5da-4d52-b696-c912e93c370a-webhook-cert\") pod \"opendatahub-operator-controller-manager-cb586fcb5-k8xhk\" (UID: \"fea8ac18-f5da-4d52-b696-c912e93c370a\") " pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:51.009033 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:51.009010 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s75bl\" (UniqueName: \"kubernetes.io/projected/fea8ac18-f5da-4d52-b696-c912e93c370a-kube-api-access-s75bl\") pod \"opendatahub-operator-controller-manager-cb586fcb5-k8xhk\" (UID: \"fea8ac18-f5da-4d52-b696-c912e93c370a\") " pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:51.088740 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:51.088641 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:51.228917 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:51.228890 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk"] Jul 22 19:01:51.231452 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:01:51.231423 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea8ac18_f5da_4d52_b696_c912e93c370a.slice/crio-aeee34fb8e0c95fe74fcd5b5f8a4b12d7d02cbe80fcb63e88d81d2ac624ebf38 WatchSource:0}: Error finding container aeee34fb8e0c95fe74fcd5b5f8a4b12d7d02cbe80fcb63e88d81d2ac624ebf38: Status 404 returned error can't find the container with id aeee34fb8e0c95fe74fcd5b5f8a4b12d7d02cbe80fcb63e88d81d2ac624ebf38 Jul 22 19:01:51.233413 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:51.233396 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 19:01:52.144255 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:52.144214 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" event={"ID":"fea8ac18-f5da-4d52-b696-c912e93c370a","Type":"ContainerStarted","Data":"aeee34fb8e0c95fe74fcd5b5f8a4b12d7d02cbe80fcb63e88d81d2ac624ebf38"} Jul 22 19:01:55.162240 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:55.162199 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" event={"ID":"fea8ac18-f5da-4d52-b696-c912e93c370a","Type":"ContainerStarted","Data":"b13fe8133bcdd3d1492afab4f40bc07fb2fd065c789ec92b95f64c1c12ab86ca"} Jul 22 19:01:56.169290 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:56.169245 2583 generic.go:356] "Generic (PLEG): container finished" podID="fea8ac18-f5da-4d52-b696-c912e93c370a" containerID="b13fe8133bcdd3d1492afab4f40bc07fb2fd065c789ec92b95f64c1c12ab86ca" exitCode=0 Jul 22 19:01:56.169737 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:56.169338 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" event={"ID":"fea8ac18-f5da-4d52-b696-c912e93c370a","Type":"ContainerDied","Data":"b13fe8133bcdd3d1492afab4f40bc07fb2fd065c789ec92b95f64c1c12ab86ca"} Jul 22 19:01:57.176025 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:57.175990 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" event={"ID":"fea8ac18-f5da-4d52-b696-c912e93c370a","Type":"ContainerStarted","Data":"a824f718579584aa760fc6b3b638ac861837356fe9be1fe6529b79e74800c19e"} Jul 22 19:01:57.176416 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:57.176135 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:01:57.199652 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:01:57.199593 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" podStartSLOduration=2.2582560210000002 podStartE2EDuration="7.199576373s" totalImagesPullingTime="3.849718325s" totalInitContainerRuntime="1.091602027s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:01:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:01:57.197579772 +0000 UTC m=+320.109743840" watchObservedRunningTime="2026-07-22 19:01:57.199576373 +0000 UTC m=+320.111740441" Jul 22 19:02:08.183554 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:08.183522 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-cb586fcb5-k8xhk" Jul 22 19:02:11.574201 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:11.574169 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-85kgq"] Jul 22 19:02:11.581704 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:11.581676 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-85kgq" Jul 22 19:02:11.587327 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:11.587307 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-l9qgm\"" Jul 22 19:02:11.587545 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:11.587532 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 22 19:02:11.601403 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:11.601374 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-85kgq"] Jul 22 19:02:11.685022 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:11.684986 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-888w9\" (UniqueName: \"kubernetes.io/projected/90c5840d-8097-4156-b044-f9b7e3e4e423-kube-api-access-888w9\") pod \"odh-model-controller-7998c67dc4-85kgq\" (UID: \"90c5840d-8097-4156-b044-f9b7e3e4e423\") " pod="opendatahub/odh-model-controller-7998c67dc4-85kgq" Jul 22 19:02:11.685198 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:11.685061 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90c5840d-8097-4156-b044-f9b7e3e4e423-cert\") pod \"odh-model-controller-7998c67dc4-85kgq\" (UID: \"90c5840d-8097-4156-b044-f9b7e3e4e423\") " pod="opendatahub/odh-model-controller-7998c67dc4-85kgq" Jul 22 19:02:11.786415 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:11.786383 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90c5840d-8097-4156-b044-f9b7e3e4e423-cert\") pod \"odh-model-controller-7998c67dc4-85kgq\" (UID: \"90c5840d-8097-4156-b044-f9b7e3e4e423\") " pod="opendatahub/odh-model-controller-7998c67dc4-85kgq" Jul 22 19:02:11.786597 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:11.786457 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-888w9\" (UniqueName: \"kubernetes.io/projected/90c5840d-8097-4156-b044-f9b7e3e4e423-kube-api-access-888w9\") pod \"odh-model-controller-7998c67dc4-85kgq\" (UID: \"90c5840d-8097-4156-b044-f9b7e3e4e423\") " pod="opendatahub/odh-model-controller-7998c67dc4-85kgq" Jul 22 19:02:11.788968 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:11.788934 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90c5840d-8097-4156-b044-f9b7e3e4e423-cert\") pod \"odh-model-controller-7998c67dc4-85kgq\" (UID: \"90c5840d-8097-4156-b044-f9b7e3e4e423\") " pod="opendatahub/odh-model-controller-7998c67dc4-85kgq" Jul 22 19:02:11.832260 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:11.832180 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-888w9\" (UniqueName: \"kubernetes.io/projected/90c5840d-8097-4156-b044-f9b7e3e4e423-kube-api-access-888w9\") pod \"odh-model-controller-7998c67dc4-85kgq\" (UID: \"90c5840d-8097-4156-b044-f9b7e3e4e423\") " pod="opendatahub/odh-model-controller-7998c67dc4-85kgq" Jul 22 19:02:11.891427 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:11.891376 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-85kgq" Jul 22 19:02:12.069137 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:12.069098 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-85kgq"] Jul 22 19:02:12.082624 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:02:12.082557 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c5840d_8097_4156_b044_f9b7e3e4e423.slice/crio-9e7a704d777665bf37e622d57101d62d06149eebf6688a24e1f6db8cc55bd2ce WatchSource:0}: Error finding container 9e7a704d777665bf37e622d57101d62d06149eebf6688a24e1f6db8cc55bd2ce: Status 404 returned error can't find the container with id 9e7a704d777665bf37e622d57101d62d06149eebf6688a24e1f6db8cc55bd2ce Jul 22 19:02:12.252217 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:12.252180 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-85kgq" event={"ID":"90c5840d-8097-4156-b044-f9b7e3e4e423","Type":"ContainerStarted","Data":"9e7a704d777665bf37e622d57101d62d06149eebf6688a24e1f6db8cc55bd2ce"} Jul 22 19:02:12.437905 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:12.437816 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-f77r6"] Jul 22 19:02:12.442393 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:12.442375 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-f77r6" Jul 22 19:02:12.445835 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:12.445812 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 22 19:02:12.454387 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:12.454361 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-f77r6"] Jul 22 19:02:12.593563 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:12.593527 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr7pf\" (UniqueName: \"kubernetes.io/projected/37f04a4f-6db5-48f7-b64d-10002f1949f5-kube-api-access-xr7pf\") pod \"model-serving-api-54d89bd5fc-f77r6\" (UID: \"37f04a4f-6db5-48f7-b64d-10002f1949f5\") " pod="opendatahub/model-serving-api-54d89bd5fc-f77r6" Jul 22 19:02:12.594034 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:12.593614 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f04a4f-6db5-48f7-b64d-10002f1949f5-tls-certs\") pod \"model-serving-api-54d89bd5fc-f77r6\" (UID: \"37f04a4f-6db5-48f7-b64d-10002f1949f5\") " pod="opendatahub/model-serving-api-54d89bd5fc-f77r6" Jul 22 19:02:12.695089 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:12.694957 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr7pf\" (UniqueName: \"kubernetes.io/projected/37f04a4f-6db5-48f7-b64d-10002f1949f5-kube-api-access-xr7pf\") pod \"model-serving-api-54d89bd5fc-f77r6\" (UID: \"37f04a4f-6db5-48f7-b64d-10002f1949f5\") " pod="opendatahub/model-serving-api-54d89bd5fc-f77r6" Jul 22 19:02:12.695317 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:12.695179 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f04a4f-6db5-48f7-b64d-10002f1949f5-tls-certs\") pod \"model-serving-api-54d89bd5fc-f77r6\" (UID: \"37f04a4f-6db5-48f7-b64d-10002f1949f5\") " pod="opendatahub/model-serving-api-54d89bd5fc-f77r6" Jul 22 19:02:12.695387 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:02:12.695316 2583 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 22 19:02:12.695437 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:02:12.695394 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37f04a4f-6db5-48f7-b64d-10002f1949f5-tls-certs podName:37f04a4f-6db5-48f7-b64d-10002f1949f5 nodeName:}" failed. No retries permitted until 2026-07-22 19:02:13.195373477 +0000 UTC m=+336.107537537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/37f04a4f-6db5-48f7-b64d-10002f1949f5-tls-certs") pod "model-serving-api-54d89bd5fc-f77r6" (UID: "37f04a4f-6db5-48f7-b64d-10002f1949f5") : secret "model-serving-api-tls" not found Jul 22 19:02:12.714393 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:12.714329 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr7pf\" (UniqueName: \"kubernetes.io/projected/37f04a4f-6db5-48f7-b64d-10002f1949f5-kube-api-access-xr7pf\") pod \"model-serving-api-54d89bd5fc-f77r6\" (UID: \"37f04a4f-6db5-48f7-b64d-10002f1949f5\") " pod="opendatahub/model-serving-api-54d89bd5fc-f77r6" Jul 22 19:02:13.201149 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:13.201024 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f04a4f-6db5-48f7-b64d-10002f1949f5-tls-certs\") pod \"model-serving-api-54d89bd5fc-f77r6\" (UID: \"37f04a4f-6db5-48f7-b64d-10002f1949f5\") " pod="opendatahub/model-serving-api-54d89bd5fc-f77r6" Jul 22 19:02:13.205046 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:13.204300 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f04a4f-6db5-48f7-b64d-10002f1949f5-tls-certs\") pod \"model-serving-api-54d89bd5fc-f77r6\" (UID: \"37f04a4f-6db5-48f7-b64d-10002f1949f5\") " pod="opendatahub/model-serving-api-54d89bd5fc-f77r6" Jul 22 19:02:13.354173 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:13.353937 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-f77r6" Jul 22 19:02:14.643240 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:14.643210 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-f77r6"] Jul 22 19:02:14.645072 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:02:14.645039 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f04a4f_6db5_48f7_b64d_10002f1949f5.slice/crio-84fa49b9649dc96da5d70a911a8374ad63c1078c542dde62aef67d48afe4ffdb WatchSource:0}: Error finding container 84fa49b9649dc96da5d70a911a8374ad63c1078c542dde62aef67d48afe4ffdb: Status 404 returned error can't find the container with id 84fa49b9649dc96da5d70a911a8374ad63c1078c542dde62aef67d48afe4ffdb Jul 22 19:02:15.275422 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:15.275372 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-f77r6" event={"ID":"37f04a4f-6db5-48f7-b64d-10002f1949f5","Type":"ContainerStarted","Data":"84fa49b9649dc96da5d70a911a8374ad63c1078c542dde62aef67d48afe4ffdb"} Jul 22 19:02:15.277267 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:15.277222 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-85kgq" event={"ID":"90c5840d-8097-4156-b044-f9b7e3e4e423","Type":"ContainerStarted","Data":"2028a7f97fdc3e8b10a612b1f151d8ed952640337ec63d4660c9f489fed19ca8"} Jul 22 19:02:15.277393 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:15.277306 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-85kgq" Jul 22 19:02:15.293514 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:15.293432 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-85kgq" podStartSLOduration=1.816243271 podStartE2EDuration="4.293410306s" totalImagesPullingTime="2.477167035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:02:15.292802612 +0000 UTC m=+338.204966686" watchObservedRunningTime="2026-07-22 19:02:15.293410306 +0000 UTC m=+338.205574374" Jul 22 19:02:17.287612 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:17.287580 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-f77r6" event={"ID":"37f04a4f-6db5-48f7-b64d-10002f1949f5","Type":"ContainerStarted","Data":"b6a98b1d6167b26c0aada4ed83d1762cdaba3ef7b2c97ed7d7855b7d12dba88d"} Jul 22 19:02:17.288010 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:17.287686 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-f77r6" Jul 22 19:02:17.308292 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:17.308237 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-f77r6" podStartSLOduration=3.645671556 podStartE2EDuration="5.308221187s" totalImagesPullingTime="1.662549631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:02:17.306817995 +0000 UTC m=+340.218982070" watchObservedRunningTime="2026-07-22 19:02:17.308221187 +0000 UTC m=+340.220385255" Jul 22 19:02:21.263997 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.263950 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h"] Jul 22 19:02:21.267362 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.267338 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4qf2d"] Jul 22 19:02:21.267522 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.267449 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h" Jul 22 19:02:21.270692 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.270666 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-87vvg\"" Jul 22 19:02:21.270822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.270725 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4qf2d" Jul 22 19:02:21.271077 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.271053 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 22 19:02:21.274128 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.274107 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kkvqb\"" Jul 22 19:02:21.274128 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.274122 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 22 19:02:21.284739 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.284711 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h"] Jul 22 19:02:21.287617 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.287590 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4qf2d"] Jul 22 19:02:21.374386 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.374352 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57bf3d99-f6f5-4763-8ca0-b2c91840d095-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4qf2d\" (UID: \"57bf3d99-f6f5-4763-8ca0-b2c91840d095\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4qf2d" Jul 22 19:02:21.374386 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.374391 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndswj\" (UniqueName: \"kubernetes.io/projected/9259a9ed-840b-4ed8-85bc-9dcb39cba2e2-kube-api-access-ndswj\") pod \"kserve-controller-manager-d6f4b65b8-5cc9h\" (UID: \"9259a9ed-840b-4ed8-85bc-9dcb39cba2e2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h" Jul 22 19:02:21.374629 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.374523 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9259a9ed-840b-4ed8-85bc-9dcb39cba2e2-cert\") pod \"kserve-controller-manager-d6f4b65b8-5cc9h\" (UID: \"9259a9ed-840b-4ed8-85bc-9dcb39cba2e2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h" Jul 22 19:02:21.374629 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.374551 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2krcn\" (UniqueName: \"kubernetes.io/projected/57bf3d99-f6f5-4763-8ca0-b2c91840d095-kube-api-access-2krcn\") pod \"llmisvc-controller-manager-777c6bd7d5-4qf2d\" (UID: \"57bf3d99-f6f5-4763-8ca0-b2c91840d095\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4qf2d" Jul 22 19:02:21.475069 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.475022 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57bf3d99-f6f5-4763-8ca0-b2c91840d095-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4qf2d\" (UID: \"57bf3d99-f6f5-4763-8ca0-b2c91840d095\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4qf2d" Jul 22 19:02:21.475069 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.475075 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndswj\" (UniqueName: \"kubernetes.io/projected/9259a9ed-840b-4ed8-85bc-9dcb39cba2e2-kube-api-access-ndswj\") pod \"kserve-controller-manager-d6f4b65b8-5cc9h\" (UID: \"9259a9ed-840b-4ed8-85bc-9dcb39cba2e2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h" Jul 22 19:02:21.475311 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.475199 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9259a9ed-840b-4ed8-85bc-9dcb39cba2e2-cert\") pod \"kserve-controller-manager-d6f4b65b8-5cc9h\" (UID: \"9259a9ed-840b-4ed8-85bc-9dcb39cba2e2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h" Jul 22 19:02:21.475311 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.475224 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2krcn\" (UniqueName: \"kubernetes.io/projected/57bf3d99-f6f5-4763-8ca0-b2c91840d095-kube-api-access-2krcn\") pod \"llmisvc-controller-manager-777c6bd7d5-4qf2d\" (UID: \"57bf3d99-f6f5-4763-8ca0-b2c91840d095\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4qf2d" Jul 22 19:02:21.475311 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:02:21.475285 2583 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 22 19:02:21.475435 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:02:21.475361 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9259a9ed-840b-4ed8-85bc-9dcb39cba2e2-cert podName:9259a9ed-840b-4ed8-85bc-9dcb39cba2e2 nodeName:}" failed. No retries permitted until 2026-07-22 19:02:21.975340712 +0000 UTC m=+344.887504773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9259a9ed-840b-4ed8-85bc-9dcb39cba2e2-cert") pod "kserve-controller-manager-d6f4b65b8-5cc9h" (UID: "9259a9ed-840b-4ed8-85bc-9dcb39cba2e2") : secret "kserve-webhook-server-cert" not found Jul 22 19:02:21.477697 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.477675 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57bf3d99-f6f5-4763-8ca0-b2c91840d095-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4qf2d\" (UID: \"57bf3d99-f6f5-4763-8ca0-b2c91840d095\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4qf2d" Jul 22 19:02:21.492022 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.491990 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndswj\" (UniqueName: \"kubernetes.io/projected/9259a9ed-840b-4ed8-85bc-9dcb39cba2e2-kube-api-access-ndswj\") pod \"kserve-controller-manager-d6f4b65b8-5cc9h\" (UID: \"9259a9ed-840b-4ed8-85bc-9dcb39cba2e2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h" Jul 22 19:02:21.492225 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.492209 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2krcn\" (UniqueName: \"kubernetes.io/projected/57bf3d99-f6f5-4763-8ca0-b2c91840d095-kube-api-access-2krcn\") pod \"llmisvc-controller-manager-777c6bd7d5-4qf2d\" (UID: \"57bf3d99-f6f5-4763-8ca0-b2c91840d095\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4qf2d" Jul 22 19:02:21.587345 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.587312 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4qf2d" Jul 22 19:02:21.756469 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.756389 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4qf2d"] Jul 22 19:02:21.756926 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:02:21.756892 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod57bf3d99_f6f5_4763_8ca0_b2c91840d095.slice/crio-8069381cc96b24ebefa182189afe286c830d365a236429fb1f343ba38b557d16 WatchSource:0}: Error finding container 8069381cc96b24ebefa182189afe286c830d365a236429fb1f343ba38b557d16: Status 404 returned error can't find the container with id 8069381cc96b24ebefa182189afe286c830d365a236429fb1f343ba38b557d16 Jul 22 19:02:21.981235 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.981150 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9259a9ed-840b-4ed8-85bc-9dcb39cba2e2-cert\") pod \"kserve-controller-manager-d6f4b65b8-5cc9h\" (UID: \"9259a9ed-840b-4ed8-85bc-9dcb39cba2e2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h" Jul 22 19:02:21.983676 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:21.983648 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9259a9ed-840b-4ed8-85bc-9dcb39cba2e2-cert\") pod \"kserve-controller-manager-d6f4b65b8-5cc9h\" (UID: \"9259a9ed-840b-4ed8-85bc-9dcb39cba2e2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h" Jul 22 19:02:22.069510 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.069466 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn"] Jul 22 19:02:22.074169 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.074149 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn" Jul 22 19:02:22.077165 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.077121 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 22 19:02:22.077343 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.077325 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-lxs7l\"" Jul 22 19:02:22.077395 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.077371 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 22 19:02:22.081732 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.081706 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1639fe19-20ec-44cd-b23b-167138c4e510-tls-certs\") pod \"kube-auth-proxy-69768f7b48-hm7nn\" (UID: \"1639fe19-20ec-44cd-b23b-167138c4e510\") " pod="openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn" Jul 22 19:02:22.081848 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.081741 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fql84\" (UniqueName: \"kubernetes.io/projected/1639fe19-20ec-44cd-b23b-167138c4e510-kube-api-access-fql84\") pod \"kube-auth-proxy-69768f7b48-hm7nn\" (UID: \"1639fe19-20ec-44cd-b23b-167138c4e510\") " pod="openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn" Jul 22 19:02:22.081848 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.081790 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1639fe19-20ec-44cd-b23b-167138c4e510-tmp\") pod \"kube-auth-proxy-69768f7b48-hm7nn\" (UID: \"1639fe19-20ec-44cd-b23b-167138c4e510\") " pod="openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn" Jul 22 19:02:22.086248 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.086228 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn"] Jul 22 19:02:22.180026 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.179991 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h" Jul 22 19:02:22.182296 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.182270 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1639fe19-20ec-44cd-b23b-167138c4e510-tmp\") pod \"kube-auth-proxy-69768f7b48-hm7nn\" (UID: \"1639fe19-20ec-44cd-b23b-167138c4e510\") " pod="openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn" Jul 22 19:02:22.182410 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.182344 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1639fe19-20ec-44cd-b23b-167138c4e510-tls-certs\") pod \"kube-auth-proxy-69768f7b48-hm7nn\" (UID: \"1639fe19-20ec-44cd-b23b-167138c4e510\") " pod="openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn" Jul 22 19:02:22.182410 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.182362 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fql84\" (UniqueName: \"kubernetes.io/projected/1639fe19-20ec-44cd-b23b-167138c4e510-kube-api-access-fql84\") pod \"kube-auth-proxy-69768f7b48-hm7nn\" (UID: \"1639fe19-20ec-44cd-b23b-167138c4e510\") " pod="openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn" Jul 22 19:02:22.182533 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:02:22.182487 2583 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 22 19:02:22.182616 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:02:22.182601 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1639fe19-20ec-44cd-b23b-167138c4e510-tls-certs podName:1639fe19-20ec-44cd-b23b-167138c4e510 nodeName:}" failed. No retries permitted until 2026-07-22 19:02:22.682581857 +0000 UTC m=+345.594745907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1639fe19-20ec-44cd-b23b-167138c4e510-tls-certs") pod "kube-auth-proxy-69768f7b48-hm7nn" (UID: "1639fe19-20ec-44cd-b23b-167138c4e510") : secret "kube-auth-proxy-tls" not found Jul 22 19:02:22.184802 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.184777 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1639fe19-20ec-44cd-b23b-167138c4e510-tmp\") pod \"kube-auth-proxy-69768f7b48-hm7nn\" (UID: \"1639fe19-20ec-44cd-b23b-167138c4e510\") " pod="openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn" Jul 22 19:02:22.201837 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.201804 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fql84\" (UniqueName: \"kubernetes.io/projected/1639fe19-20ec-44cd-b23b-167138c4e510-kube-api-access-fql84\") pod \"kube-auth-proxy-69768f7b48-hm7nn\" (UID: \"1639fe19-20ec-44cd-b23b-167138c4e510\") " pod="openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn" Jul 22 19:02:22.314215 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.314179 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4qf2d" event={"ID":"57bf3d99-f6f5-4763-8ca0-b2c91840d095","Type":"ContainerStarted","Data":"8069381cc96b24ebefa182189afe286c830d365a236429fb1f343ba38b557d16"} Jul 22 19:02:22.327239 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.327214 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h"] Jul 22 19:02:22.329239 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:02:22.329211 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9259a9ed_840b_4ed8_85bc_9dcb39cba2e2.slice/crio-ef31933c4f879398a80a2815d79b4fa68e6c73549c5da546592668198b31c2c1 WatchSource:0}: Error finding container ef31933c4f879398a80a2815d79b4fa68e6c73549c5da546592668198b31c2c1: Status 404 returned error can't find the container with id ef31933c4f879398a80a2815d79b4fa68e6c73549c5da546592668198b31c2c1 Jul 22 19:02:22.686987 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.686950 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1639fe19-20ec-44cd-b23b-167138c4e510-tls-certs\") pod \"kube-auth-proxy-69768f7b48-hm7nn\" (UID: \"1639fe19-20ec-44cd-b23b-167138c4e510\") " pod="openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn" Jul 22 19:02:22.689369 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.689344 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1639fe19-20ec-44cd-b23b-167138c4e510-tls-certs\") pod \"kube-auth-proxy-69768f7b48-hm7nn\" (UID: \"1639fe19-20ec-44cd-b23b-167138c4e510\") " pod="openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn" Jul 22 19:02:22.984918 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:22.984845 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn" Jul 22 19:02:23.229153 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:23.229120 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn"] Jul 22 19:02:23.231755 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:02:23.231723 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1639fe19_20ec_44cd_b23b_167138c4e510.slice/crio-fd96f8f22477f2a2115efca3ffab868fd78936f506b1b6b1bfc4ef9781d00837 WatchSource:0}: Error finding container fd96f8f22477f2a2115efca3ffab868fd78936f506b1b6b1bfc4ef9781d00837: Status 404 returned error can't find the container with id fd96f8f22477f2a2115efca3ffab868fd78936f506b1b6b1bfc4ef9781d00837 Jul 22 19:02:23.322120 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:23.322077 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn" event={"ID":"1639fe19-20ec-44cd-b23b-167138c4e510","Type":"ContainerStarted","Data":"fd96f8f22477f2a2115efca3ffab868fd78936f506b1b6b1bfc4ef9781d00837"} Jul 22 19:02:23.323654 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:23.323620 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h" event={"ID":"9259a9ed-840b-4ed8-85bc-9dcb39cba2e2","Type":"ContainerStarted","Data":"ef31933c4f879398a80a2815d79b4fa68e6c73549c5da546592668198b31c2c1"} Jul 22 19:02:24.428842 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.428795 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t"] Jul 22 19:02:24.436612 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.436581 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.440526 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.440370 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 19:02:24.440526 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.440386 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 19:02:24.441557 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.441534 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kck5d\"" Jul 22 19:02:24.441557 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.441552 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 19:02:24.448687 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.448664 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 19:02:24.476028 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.475996 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t"] Jul 22 19:02:24.500462 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.500426 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.500645 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.500513 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.500645 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.500562 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4cn\" (UniqueName: \"kubernetes.io/projected/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-kube-api-access-fz4cn\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.500775 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.500659 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.500775 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.500734 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.500872 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.500771 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.500872 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.500798 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.601913 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.601874 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.602098 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.601925 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4cn\" (UniqueName: \"kubernetes.io/projected/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-kube-api-access-fz4cn\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.602098 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.602006 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.602098 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.602071 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.602268 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.602102 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.602268 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.602129 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.602268 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.602164 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.604206 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.604176 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.606316 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.606282 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.606853 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.606831 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.606947 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.606913 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.607397 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.607375 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.631044 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.630977 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4cn\" (UniqueName: \"kubernetes.io/projected/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-kube-api-access-fz4cn\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.642926 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.642864 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6eb0a8d-45b6-4f82-af15-5e714fac5c52-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-kc42t\" (UID: \"b6eb0a8d-45b6-4f82-af15-5e714fac5c52\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:24.751555 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:24.751459 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:26.283651 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:26.283611 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-85kgq" Jul 22 19:02:26.309617 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:26.309587 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t"] Jul 22 19:02:26.313358 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:02:26.313186 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6eb0a8d_45b6_4f82_af15_5e714fac5c52.slice/crio-df74bf0e3a98beae3c7964678cb516d2f92093571687f8a32aa2e6318c3f4b8c WatchSource:0}: Error finding container df74bf0e3a98beae3c7964678cb516d2f92093571687f8a32aa2e6318c3f4b8c: Status 404 returned error can't find the container with id df74bf0e3a98beae3c7964678cb516d2f92093571687f8a32aa2e6318c3f4b8c Jul 22 19:02:26.358537 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:26.356483 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4qf2d" event={"ID":"57bf3d99-f6f5-4763-8ca0-b2c91840d095","Type":"ContainerStarted","Data":"710a50b1ecef2bdb8057be33a85852fe1e98585e3b4c638fbab9ecaad6f4eaa8"} Jul 22 19:02:26.358537 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:26.356585 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4qf2d" Jul 22 19:02:26.359413 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:26.359383 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" event={"ID":"b6eb0a8d-45b6-4f82-af15-5e714fac5c52","Type":"ContainerStarted","Data":"df74bf0e3a98beae3c7964678cb516d2f92093571687f8a32aa2e6318c3f4b8c"} Jul 22 19:02:26.361894 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:26.361859 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h" event={"ID":"9259a9ed-840b-4ed8-85bc-9dcb39cba2e2","Type":"ContainerStarted","Data":"c8f6b54299b4c03dcd7dc67d30634c194abd1fb403b43c4f525c10e4a0deb2a4"} Jul 22 19:02:26.362352 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:26.362180 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h" Jul 22 19:02:26.364102 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:26.364079 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn" event={"ID":"1639fe19-20ec-44cd-b23b-167138c4e510","Type":"ContainerStarted","Data":"ca07eb9daf3b6be1c86e6c696773f1f247a3e9d33d4bacefd43e095adf6172f2"} Jul 22 19:02:26.382485 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:26.382425 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4qf2d" podStartSLOduration=0.980888703 podStartE2EDuration="5.382405415s" totalImagesPullingTime="4.401516712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:02:26.380809355 +0000 UTC m=+349.292973422" watchObservedRunningTime="2026-07-22 19:02:26.382405415 +0000 UTC m=+349.294569464" Jul 22 19:02:26.422703 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:26.422599 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69768f7b48-hm7nn" podStartSLOduration=1.437508481 podStartE2EDuration="4.422581713s" totalImagesPullingTime="2.985073232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:02:26.421786689 +0000 UTC m=+349.333950768" watchObservedRunningTime="2026-07-22 19:02:26.422581713 +0000 UTC m=+349.334745780" Jul 22 19:02:26.423587 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:26.423558 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h" podStartSLOduration=1.591310274 podStartE2EDuration="5.423549342s" totalImagesPullingTime="3.832239068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:02:26.40124768 +0000 UTC m=+349.313411749" watchObservedRunningTime="2026-07-22 19:02:26.423549342 +0000 UTC m=+349.335713411" Jul 22 19:02:28.298185 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:28.298094 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-f77r6" Jul 22 19:02:28.803452 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:28.803411 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 19:02:28.803561 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:28.803484 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 19:02:29.385082 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:29.385041 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" event={"ID":"b6eb0a8d-45b6-4f82-af15-5e714fac5c52","Type":"ContainerStarted","Data":"36b75cc543375a24b41c48fc84b0689d1f8cb88919c4ba0e38c274b09e9e3b81"} Jul 22 19:02:29.385798 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:29.385748 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:29.387510 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:29.387465 2583 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-kc42t container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 19:02:29.387634 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:29.387538 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" podUID="b6eb0a8d-45b6-4f82-af15-5e714fac5c52" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 19:02:29.411251 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:29.411191 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" podStartSLOduration=2.923764093 podStartE2EDuration="5.41117498s" totalImagesPullingTime="2.487410887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:02:29.408766986 +0000 UTC m=+352.320931055" watchObservedRunningTime="2026-07-22 19:02:29.41117498 +0000 UTC m=+352.323339047" Jul 22 19:02:30.391641 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:30.391612 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kc42t" Jul 22 19:02:57.373792 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:57.373762 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4qf2d" Jul 22 19:02:57.376696 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:02:57.376677 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cc9h" Jul 22 19:03:27.768726 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:27.768692 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn"] Jul 22 19:03:27.776000 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:27.775973 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" Jul 22 19:03:27.779390 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:27.779189 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 19:03:27.780473 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:27.780442 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 19:03:27.784012 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:27.780994 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7mkgk\"" Jul 22 19:03:27.784012 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:27.783847 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn"] Jul 22 19:03:27.868436 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:27.868396 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd614366-354d-4198-a00b-c4bad2d27b92-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn\" (UID: \"fd614366-354d-4198-a00b-c4bad2d27b92\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" Jul 22 19:03:27.868640 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:27.868553 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd614366-354d-4198-a00b-c4bad2d27b92-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn\" (UID: \"fd614366-354d-4198-a00b-c4bad2d27b92\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" Jul 22 19:03:27.868640 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:27.868591 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68jkl\" (UniqueName: \"kubernetes.io/projected/fd614366-354d-4198-a00b-c4bad2d27b92-kube-api-access-68jkl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn\" (UID: \"fd614366-354d-4198-a00b-c4bad2d27b92\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" Jul 22 19:03:27.969294 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:27.969253 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd614366-354d-4198-a00b-c4bad2d27b92-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn\" (UID: \"fd614366-354d-4198-a00b-c4bad2d27b92\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" Jul 22 19:03:27.969469 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:27.969364 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd614366-354d-4198-a00b-c4bad2d27b92-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn\" (UID: \"fd614366-354d-4198-a00b-c4bad2d27b92\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" Jul 22 19:03:27.969469 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:27.969393 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68jkl\" (UniqueName: \"kubernetes.io/projected/fd614366-354d-4198-a00b-c4bad2d27b92-kube-api-access-68jkl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn\" (UID: \"fd614366-354d-4198-a00b-c4bad2d27b92\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" Jul 22 19:03:27.969669 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:27.969649 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd614366-354d-4198-a00b-c4bad2d27b92-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn\" (UID: \"fd614366-354d-4198-a00b-c4bad2d27b92\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" Jul 22 19:03:27.969753 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:27.969732 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd614366-354d-4198-a00b-c4bad2d27b92-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn\" (UID: \"fd614366-354d-4198-a00b-c4bad2d27b92\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" Jul 22 19:03:27.978272 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:27.978245 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68jkl\" (UniqueName: \"kubernetes.io/projected/fd614366-354d-4198-a00b-c4bad2d27b92-kube-api-access-68jkl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn\" (UID: \"fd614366-354d-4198-a00b-c4bad2d27b92\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" Jul 22 19:03:28.094117 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.094024 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" Jul 22 19:03:28.252857 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.252831 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn"] Jul 22 19:03:28.254578 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:03:28.254554 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd614366_354d_4198_a00b_c4bad2d27b92.slice/crio-0a56ac20f888c5a5e278a2b76f5cb25c35ec38bc282af92475ba5e06b3a3b75b WatchSource:0}: Error finding container 0a56ac20f888c5a5e278a2b76f5cb25c35ec38bc282af92475ba5e06b3a3b75b: Status 404 returned error can't find the container with id 0a56ac20f888c5a5e278a2b76f5cb25c35ec38bc282af92475ba5e06b3a3b75b Jul 22 19:03:28.374253 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.374167 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6"] Jul 22 19:03:28.376990 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.376948 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" Jul 22 19:03:28.386515 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.386464 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6"] Jul 22 19:03:28.473463 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.473428 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65vhn\" (UniqueName: \"kubernetes.io/projected/fc860783-da97-4027-bce5-75d91c7dd87d-kube-api-access-65vhn\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6\" (UID: \"fc860783-da97-4027-bce5-75d91c7dd87d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" Jul 22 19:03:28.473675 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.473552 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc860783-da97-4027-bce5-75d91c7dd87d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6\" (UID: \"fc860783-da97-4027-bce5-75d91c7dd87d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" Jul 22 19:03:28.473675 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.473582 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc860783-da97-4027-bce5-75d91c7dd87d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6\" (UID: \"fc860783-da97-4027-bce5-75d91c7dd87d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" Jul 22 19:03:28.574823 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.574784 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc860783-da97-4027-bce5-75d91c7dd87d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6\" (UID: \"fc860783-da97-4027-bce5-75d91c7dd87d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" Jul 22 19:03:28.574823 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.574823 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc860783-da97-4027-bce5-75d91c7dd87d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6\" (UID: \"fc860783-da97-4027-bce5-75d91c7dd87d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" Jul 22 19:03:28.574996 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.574869 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65vhn\" (UniqueName: \"kubernetes.io/projected/fc860783-da97-4027-bce5-75d91c7dd87d-kube-api-access-65vhn\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6\" (UID: \"fc860783-da97-4027-bce5-75d91c7dd87d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" Jul 22 19:03:28.575277 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.575255 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc860783-da97-4027-bce5-75d91c7dd87d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6\" (UID: \"fc860783-da97-4027-bce5-75d91c7dd87d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" Jul 22 19:03:28.575313 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.575269 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc860783-da97-4027-bce5-75d91c7dd87d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6\" (UID: \"fc860783-da97-4027-bce5-75d91c7dd87d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" Jul 22 19:03:28.584187 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.584156 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65vhn\" (UniqueName: \"kubernetes.io/projected/fc860783-da97-4027-bce5-75d91c7dd87d-kube-api-access-65vhn\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6\" (UID: \"fc860783-da97-4027-bce5-75d91c7dd87d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" Jul 22 19:03:28.694261 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.694176 2583 generic.go:356] "Generic (PLEG): container finished" podID="fd614366-354d-4198-a00b-c4bad2d27b92" containerID="07a6be1e59c2c3417e36f65a3390b3e82a28caeba9769cba139a6a1df3f4e7c6" exitCode=0 Jul 22 19:03:28.694399 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.694262 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" event={"ID":"fd614366-354d-4198-a00b-c4bad2d27b92","Type":"ContainerDied","Data":"07a6be1e59c2c3417e36f65a3390b3e82a28caeba9769cba139a6a1df3f4e7c6"} Jul 22 19:03:28.694399 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.694314 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" event={"ID":"fd614366-354d-4198-a00b-c4bad2d27b92","Type":"ContainerStarted","Data":"0a56ac20f888c5a5e278a2b76f5cb25c35ec38bc282af92475ba5e06b3a3b75b"} Jul 22 19:03:28.720149 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.720114 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" Jul 22 19:03:28.784798 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.784768 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd"] Jul 22 19:03:28.788036 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.788015 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" Jul 22 19:03:28.799162 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.799130 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd"] Jul 22 19:03:28.865299 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.865273 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6"] Jul 22 19:03:28.866637 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:03:28.866611 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc860783_da97_4027_bce5_75d91c7dd87d.slice/crio-3aac371439512d5ee06f6ceba7bd726889c57041efc4da1eaffe29cf90136050 WatchSource:0}: Error finding container 3aac371439512d5ee06f6ceba7bd726889c57041efc4da1eaffe29cf90136050: Status 404 returned error can't find the container with id 3aac371439512d5ee06f6ceba7bd726889c57041efc4da1eaffe29cf90136050 Jul 22 19:03:28.877736 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.877710 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6371b6ea-2108-4ab2-8d67-8ded5127613a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd\" (UID: \"6371b6ea-2108-4ab2-8d67-8ded5127613a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" Jul 22 19:03:28.877827 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.877758 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptrv7\" (UniqueName: \"kubernetes.io/projected/6371b6ea-2108-4ab2-8d67-8ded5127613a-kube-api-access-ptrv7\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd\" (UID: \"6371b6ea-2108-4ab2-8d67-8ded5127613a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" Jul 22 19:03:28.877885 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.877822 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6371b6ea-2108-4ab2-8d67-8ded5127613a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd\" (UID: \"6371b6ea-2108-4ab2-8d67-8ded5127613a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" Jul 22 19:03:28.978360 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.978326 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6371b6ea-2108-4ab2-8d67-8ded5127613a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd\" (UID: \"6371b6ea-2108-4ab2-8d67-8ded5127613a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" Jul 22 19:03:28.978531 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.978377 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptrv7\" (UniqueName: \"kubernetes.io/projected/6371b6ea-2108-4ab2-8d67-8ded5127613a-kube-api-access-ptrv7\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd\" (UID: \"6371b6ea-2108-4ab2-8d67-8ded5127613a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" Jul 22 19:03:28.978531 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.978405 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6371b6ea-2108-4ab2-8d67-8ded5127613a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd\" (UID: \"6371b6ea-2108-4ab2-8d67-8ded5127613a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" Jul 22 19:03:28.978796 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.978774 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6371b6ea-2108-4ab2-8d67-8ded5127613a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd\" (UID: \"6371b6ea-2108-4ab2-8d67-8ded5127613a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" Jul 22 19:03:28.978869 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.978782 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6371b6ea-2108-4ab2-8d67-8ded5127613a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd\" (UID: \"6371b6ea-2108-4ab2-8d67-8ded5127613a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" Jul 22 19:03:28.987907 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:28.987887 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptrv7\" (UniqueName: \"kubernetes.io/projected/6371b6ea-2108-4ab2-8d67-8ded5127613a-kube-api-access-ptrv7\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd\" (UID: \"6371b6ea-2108-4ab2-8d67-8ded5127613a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" Jul 22 19:03:29.098550 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.098516 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" Jul 22 19:03:29.187827 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.187795 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq"] Jul 22 19:03:29.190930 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.190910 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" Jul 22 19:03:29.201153 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.201128 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq"] Jul 22 19:03:29.241834 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.241755 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd"] Jul 22 19:03:29.244760 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:03:29.244732 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6371b6ea_2108_4ab2_8d67_8ded5127613a.slice/crio-4abd8953dceb4576975aec7177b8a393e0607ee56cbfdf606bab62cda7818deb WatchSource:0}: Error finding container 4abd8953dceb4576975aec7177b8a393e0607ee56cbfdf606bab62cda7818deb: Status 404 returned error can't find the container with id 4abd8953dceb4576975aec7177b8a393e0607ee56cbfdf606bab62cda7818deb Jul 22 19:03:29.280979 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.280951 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/334f8642-4498-4c8c-9c13-d810152519fa-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq\" (UID: \"334f8642-4498-4c8c-9c13-d810152519fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" Jul 22 19:03:29.281114 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.281034 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/334f8642-4498-4c8c-9c13-d810152519fa-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq\" (UID: \"334f8642-4498-4c8c-9c13-d810152519fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" Jul 22 19:03:29.281114 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.281082 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfdwj\" (UniqueName: \"kubernetes.io/projected/334f8642-4498-4c8c-9c13-d810152519fa-kube-api-access-mfdwj\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq\" (UID: \"334f8642-4498-4c8c-9c13-d810152519fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" Jul 22 19:03:29.382176 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.382135 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/334f8642-4498-4c8c-9c13-d810152519fa-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq\" (UID: \"334f8642-4498-4c8c-9c13-d810152519fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" Jul 22 19:03:29.382324 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.382221 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/334f8642-4498-4c8c-9c13-d810152519fa-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq\" (UID: \"334f8642-4498-4c8c-9c13-d810152519fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" Jul 22 19:03:29.382324 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.382283 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfdwj\" (UniqueName: \"kubernetes.io/projected/334f8642-4498-4c8c-9c13-d810152519fa-kube-api-access-mfdwj\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq\" (UID: \"334f8642-4498-4c8c-9c13-d810152519fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" Jul 22 19:03:29.382535 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.382484 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/334f8642-4498-4c8c-9c13-d810152519fa-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq\" (UID: \"334f8642-4498-4c8c-9c13-d810152519fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" Jul 22 19:03:29.382693 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.382595 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/334f8642-4498-4c8c-9c13-d810152519fa-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq\" (UID: \"334f8642-4498-4c8c-9c13-d810152519fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" Jul 22 19:03:29.398982 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.398954 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfdwj\" (UniqueName: \"kubernetes.io/projected/334f8642-4498-4c8c-9c13-d810152519fa-kube-api-access-mfdwj\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq\" (UID: \"334f8642-4498-4c8c-9c13-d810152519fa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" Jul 22 19:03:29.502263 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.502239 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" Jul 22 19:03:29.647768 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.647733 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq"] Jul 22 19:03:29.660426 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:03:29.660402 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334f8642_4498_4c8c_9c13_d810152519fa.slice/crio-0c3943085be8035a470c596ac9568548ea803d39f582bcc989d1472485801d69 WatchSource:0}: Error finding container 0c3943085be8035a470c596ac9568548ea803d39f582bcc989d1472485801d69: Status 404 returned error can't find the container with id 0c3943085be8035a470c596ac9568548ea803d39f582bcc989d1472485801d69 Jul 22 19:03:29.700875 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.700847 2583 generic.go:356] "Generic (PLEG): container finished" podID="fc860783-da97-4027-bce5-75d91c7dd87d" containerID="b40d481cc81ff24031b0ffc06e11e1ee1d15165a5844b306ab7064428a1c16a4" exitCode=0 Jul 22 19:03:29.701024 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.700942 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" event={"ID":"fc860783-da97-4027-bce5-75d91c7dd87d","Type":"ContainerDied","Data":"b40d481cc81ff24031b0ffc06e11e1ee1d15165a5844b306ab7064428a1c16a4"} Jul 22 19:03:29.701024 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.701007 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" event={"ID":"fc860783-da97-4027-bce5-75d91c7dd87d","Type":"ContainerStarted","Data":"3aac371439512d5ee06f6ceba7bd726889c57041efc4da1eaffe29cf90136050"} Jul 22 19:03:29.703053 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.703028 2583 generic.go:356] "Generic (PLEG): container finished" podID="fd614366-354d-4198-a00b-c4bad2d27b92" containerID="5de38301f10fe458ebf557d63af82051abf4eaaa3649995691f56628aa2441ba" exitCode=0 Jul 22 19:03:29.703172 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.703112 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" event={"ID":"fd614366-354d-4198-a00b-c4bad2d27b92","Type":"ContainerDied","Data":"5de38301f10fe458ebf557d63af82051abf4eaaa3649995691f56628aa2441ba"} Jul 22 19:03:29.704650 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.704624 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" event={"ID":"334f8642-4498-4c8c-9c13-d810152519fa","Type":"ContainerStarted","Data":"0c3943085be8035a470c596ac9568548ea803d39f582bcc989d1472485801d69"} Jul 22 19:03:29.706316 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.706294 2583 generic.go:356] "Generic (PLEG): container finished" podID="6371b6ea-2108-4ab2-8d67-8ded5127613a" containerID="ed4f511f6c1de819b32dc43aad9af0731064811a5af86d9a56f166db0d00a0ef" exitCode=0 Jul 22 19:03:29.706435 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.706355 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" event={"ID":"6371b6ea-2108-4ab2-8d67-8ded5127613a","Type":"ContainerDied","Data":"ed4f511f6c1de819b32dc43aad9af0731064811a5af86d9a56f166db0d00a0ef"} Jul 22 19:03:29.706435 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:29.706384 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" event={"ID":"6371b6ea-2108-4ab2-8d67-8ded5127613a","Type":"ContainerStarted","Data":"4abd8953dceb4576975aec7177b8a393e0607ee56cbfdf606bab62cda7818deb"} Jul 22 19:03:30.713973 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:30.713941 2583 generic.go:356] "Generic (PLEG): container finished" podID="fc860783-da97-4027-bce5-75d91c7dd87d" containerID="3949aa4e4a7768c6b8fdf788fce06f9c4188d4974b2e32c36734be4c224af3a2" exitCode=0 Jul 22 19:03:30.714428 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:30.714032 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" event={"ID":"fc860783-da97-4027-bce5-75d91c7dd87d","Type":"ContainerDied","Data":"3949aa4e4a7768c6b8fdf788fce06f9c4188d4974b2e32c36734be4c224af3a2"} Jul 22 19:03:30.716395 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:30.716362 2583 generic.go:356] "Generic (PLEG): container finished" podID="fd614366-354d-4198-a00b-c4bad2d27b92" containerID="5ef69e25671c49c2724b179e5711f293bbd0fc862b150a3191545bc715a4a266" exitCode=0 Jul 22 19:03:30.716512 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:30.716438 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" event={"ID":"fd614366-354d-4198-a00b-c4bad2d27b92","Type":"ContainerDied","Data":"5ef69e25671c49c2724b179e5711f293bbd0fc862b150a3191545bc715a4a266"} Jul 22 19:03:30.717977 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:30.717952 2583 generic.go:356] "Generic (PLEG): container finished" podID="334f8642-4498-4c8c-9c13-d810152519fa" containerID="1c88f8b40259c269797ec6809e9487b5835356eaa743822a669fc12ddfac911d" exitCode=0 Jul 22 19:03:30.718092 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:30.718073 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" event={"ID":"334f8642-4498-4c8c-9c13-d810152519fa","Type":"ContainerDied","Data":"1c88f8b40259c269797ec6809e9487b5835356eaa743822a669fc12ddfac911d"} Jul 22 19:03:30.719935 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:30.719915 2583 generic.go:356] "Generic (PLEG): container finished" podID="6371b6ea-2108-4ab2-8d67-8ded5127613a" containerID="d9b007654a6d365a059ca66a8b40d52d37561f281f51fe5d02450759bb5597ab" exitCode=0 Jul 22 19:03:30.720047 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:30.719943 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" event={"ID":"6371b6ea-2108-4ab2-8d67-8ded5127613a","Type":"ContainerDied","Data":"d9b007654a6d365a059ca66a8b40d52d37561f281f51fe5d02450759bb5597ab"} Jul 22 19:03:31.727996 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:31.727908 2583 generic.go:356] "Generic (PLEG): container finished" podID="6371b6ea-2108-4ab2-8d67-8ded5127613a" containerID="cf25de0d28eec35a7186511d91fb7eb3bcfd37669b5d35eb57cb4c9e20896fb2" exitCode=0 Jul 22 19:03:31.728405 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:31.727987 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" event={"ID":"6371b6ea-2108-4ab2-8d67-8ded5127613a","Type":"ContainerDied","Data":"cf25de0d28eec35a7186511d91fb7eb3bcfd37669b5d35eb57cb4c9e20896fb2"} Jul 22 19:03:31.730132 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:31.730110 2583 generic.go:356] "Generic (PLEG): container finished" podID="fc860783-da97-4027-bce5-75d91c7dd87d" containerID="14b9b96ee3b3ee833b9d76f5fa1a1269c33ac4e1b30edc7572599986b34929fa" exitCode=0 Jul 22 19:03:31.730306 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:31.730189 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" event={"ID":"fc860783-da97-4027-bce5-75d91c7dd87d","Type":"ContainerDied","Data":"14b9b96ee3b3ee833b9d76f5fa1a1269c33ac4e1b30edc7572599986b34929fa"} Jul 22 19:03:31.731947 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:31.731922 2583 generic.go:356] "Generic (PLEG): container finished" podID="334f8642-4498-4c8c-9c13-d810152519fa" containerID="cf3dbe5ce7fa09c0f7663e95ccee9c12a0d5d1c9c25a818c6132b4aa202fe8f5" exitCode=0 Jul 22 19:03:31.732064 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:31.732013 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" event={"ID":"334f8642-4498-4c8c-9c13-d810152519fa","Type":"ContainerDied","Data":"cf3dbe5ce7fa09c0f7663e95ccee9c12a0d5d1c9c25a818c6132b4aa202fe8f5"} Jul 22 19:03:31.870664 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:31.870640 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" Jul 22 19:03:32.007358 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:32.007323 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd614366-354d-4198-a00b-c4bad2d27b92-bundle\") pod \"fd614366-354d-4198-a00b-c4bad2d27b92\" (UID: \"fd614366-354d-4198-a00b-c4bad2d27b92\") " Jul 22 19:03:32.007526 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:32.007398 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd614366-354d-4198-a00b-c4bad2d27b92-util\") pod \"fd614366-354d-4198-a00b-c4bad2d27b92\" (UID: \"fd614366-354d-4198-a00b-c4bad2d27b92\") " Jul 22 19:03:32.007526 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:32.007428 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68jkl\" (UniqueName: \"kubernetes.io/projected/fd614366-354d-4198-a00b-c4bad2d27b92-kube-api-access-68jkl\") pod \"fd614366-354d-4198-a00b-c4bad2d27b92\" (UID: \"fd614366-354d-4198-a00b-c4bad2d27b92\") " Jul 22 19:03:32.007898 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:32.007868 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd614366-354d-4198-a00b-c4bad2d27b92-bundle" (OuterVolumeSpecName: "bundle") pod "fd614366-354d-4198-a00b-c4bad2d27b92" (UID: "fd614366-354d-4198-a00b-c4bad2d27b92"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:03:32.009871 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:32.009842 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd614366-354d-4198-a00b-c4bad2d27b92-kube-api-access-68jkl" (OuterVolumeSpecName: "kube-api-access-68jkl") pod "fd614366-354d-4198-a00b-c4bad2d27b92" (UID: "fd614366-354d-4198-a00b-c4bad2d27b92"). InnerVolumeSpecName "kube-api-access-68jkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:03:32.013250 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:32.013205 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd614366-354d-4198-a00b-c4bad2d27b92-util" (OuterVolumeSpecName: "util") pod "fd614366-354d-4198-a00b-c4bad2d27b92" (UID: "fd614366-354d-4198-a00b-c4bad2d27b92"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:03:32.108959 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:32.108921 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd614366-354d-4198-a00b-c4bad2d27b92-util\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:03:32.108959 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:32.108952 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68jkl\" (UniqueName: \"kubernetes.io/projected/fd614366-354d-4198-a00b-c4bad2d27b92-kube-api-access-68jkl\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:03:32.108959 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:32.108961 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd614366-354d-4198-a00b-c4bad2d27b92-bundle\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:03:32.739217 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:32.739183 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" Jul 22 19:03:32.739642 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:32.739190 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn" event={"ID":"fd614366-354d-4198-a00b-c4bad2d27b92","Type":"ContainerDied","Data":"0a56ac20f888c5a5e278a2b76f5cb25c35ec38bc282af92475ba5e06b3a3b75b"} Jul 22 19:03:32.739642 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:32.739324 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a56ac20f888c5a5e278a2b76f5cb25c35ec38bc282af92475ba5e06b3a3b75b" Jul 22 19:03:32.741260 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:32.741231 2583 generic.go:356] "Generic (PLEG): container finished" podID="334f8642-4498-4c8c-9c13-d810152519fa" containerID="994c3826c3e2dc380681a275845697f3a59eb5888c07bfc70aef49ae91ce52b6" exitCode=0 Jul 22 19:03:32.741384 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:32.741303 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" event={"ID":"334f8642-4498-4c8c-9c13-d810152519fa","Type":"ContainerDied","Data":"994c3826c3e2dc380681a275845697f3a59eb5888c07bfc70aef49ae91ce52b6"} Jul 22 19:03:32.881174 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:32.881151 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" Jul 22 19:03:32.909799 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:32.909760 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" Jul 22 19:03:33.017796 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.017773 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc860783-da97-4027-bce5-75d91c7dd87d-bundle\") pod \"fc860783-da97-4027-bce5-75d91c7dd87d\" (UID: \"fc860783-da97-4027-bce5-75d91c7dd87d\") " Jul 22 19:03:33.017930 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.017842 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6371b6ea-2108-4ab2-8d67-8ded5127613a-util\") pod \"6371b6ea-2108-4ab2-8d67-8ded5127613a\" (UID: \"6371b6ea-2108-4ab2-8d67-8ded5127613a\") " Jul 22 19:03:33.017930 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.017877 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6371b6ea-2108-4ab2-8d67-8ded5127613a-bundle\") pod \"6371b6ea-2108-4ab2-8d67-8ded5127613a\" (UID: \"6371b6ea-2108-4ab2-8d67-8ded5127613a\") " Jul 22 19:03:33.017930 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.017893 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptrv7\" (UniqueName: \"kubernetes.io/projected/6371b6ea-2108-4ab2-8d67-8ded5127613a-kube-api-access-ptrv7\") pod \"6371b6ea-2108-4ab2-8d67-8ded5127613a\" (UID: \"6371b6ea-2108-4ab2-8d67-8ded5127613a\") " Jul 22 19:03:33.018099 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.017950 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65vhn\" (UniqueName: \"kubernetes.io/projected/fc860783-da97-4027-bce5-75d91c7dd87d-kube-api-access-65vhn\") pod \"fc860783-da97-4027-bce5-75d91c7dd87d\" (UID: \"fc860783-da97-4027-bce5-75d91c7dd87d\") " Jul 22 19:03:33.018099 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.017965 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc860783-da97-4027-bce5-75d91c7dd87d-util\") pod \"fc860783-da97-4027-bce5-75d91c7dd87d\" (UID: \"fc860783-da97-4027-bce5-75d91c7dd87d\") " Jul 22 19:03:33.018456 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.018421 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6371b6ea-2108-4ab2-8d67-8ded5127613a-bundle" (OuterVolumeSpecName: "bundle") pod "6371b6ea-2108-4ab2-8d67-8ded5127613a" (UID: "6371b6ea-2108-4ab2-8d67-8ded5127613a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:03:33.018655 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.018628 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc860783-da97-4027-bce5-75d91c7dd87d-bundle" (OuterVolumeSpecName: "bundle") pod "fc860783-da97-4027-bce5-75d91c7dd87d" (UID: "fc860783-da97-4027-bce5-75d91c7dd87d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:03:33.020239 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.020215 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc860783-da97-4027-bce5-75d91c7dd87d-kube-api-access-65vhn" (OuterVolumeSpecName: "kube-api-access-65vhn") pod "fc860783-da97-4027-bce5-75d91c7dd87d" (UID: "fc860783-da97-4027-bce5-75d91c7dd87d"). InnerVolumeSpecName "kube-api-access-65vhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:03:33.020325 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.020215 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6371b6ea-2108-4ab2-8d67-8ded5127613a-kube-api-access-ptrv7" (OuterVolumeSpecName: "kube-api-access-ptrv7") pod "6371b6ea-2108-4ab2-8d67-8ded5127613a" (UID: "6371b6ea-2108-4ab2-8d67-8ded5127613a"). InnerVolumeSpecName "kube-api-access-ptrv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:03:33.023015 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.022980 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc860783-da97-4027-bce5-75d91c7dd87d-util" (OuterVolumeSpecName: "util") pod "fc860783-da97-4027-bce5-75d91c7dd87d" (UID: "fc860783-da97-4027-bce5-75d91c7dd87d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:03:33.026449 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.026424 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6371b6ea-2108-4ab2-8d67-8ded5127613a-util" (OuterVolumeSpecName: "util") pod "6371b6ea-2108-4ab2-8d67-8ded5127613a" (UID: "6371b6ea-2108-4ab2-8d67-8ded5127613a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:03:33.119616 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.119580 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc860783-da97-4027-bce5-75d91c7dd87d-bundle\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:03:33.119616 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.119611 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6371b6ea-2108-4ab2-8d67-8ded5127613a-util\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:03:33.119616 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.119620 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6371b6ea-2108-4ab2-8d67-8ded5127613a-bundle\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:03:33.119910 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.119628 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptrv7\" (UniqueName: \"kubernetes.io/projected/6371b6ea-2108-4ab2-8d67-8ded5127613a-kube-api-access-ptrv7\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:03:33.119910 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.119639 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65vhn\" (UniqueName: \"kubernetes.io/projected/fc860783-da97-4027-bce5-75d91c7dd87d-kube-api-access-65vhn\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:03:33.119910 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.119649 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc860783-da97-4027-bce5-75d91c7dd87d-util\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:03:33.749084 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.749048 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" Jul 22 19:03:33.749558 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.749081 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6" event={"ID":"fc860783-da97-4027-bce5-75d91c7dd87d","Type":"ContainerDied","Data":"3aac371439512d5ee06f6ceba7bd726889c57041efc4da1eaffe29cf90136050"} Jul 22 19:03:33.749558 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.749134 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3aac371439512d5ee06f6ceba7bd726889c57041efc4da1eaffe29cf90136050" Jul 22 19:03:33.751068 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.751039 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" event={"ID":"6371b6ea-2108-4ab2-8d67-8ded5127613a","Type":"ContainerDied","Data":"4abd8953dceb4576975aec7177b8a393e0607ee56cbfdf606bab62cda7818deb"} Jul 22 19:03:33.751201 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.751072 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd" Jul 22 19:03:33.751201 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.751083 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4abd8953dceb4576975aec7177b8a393e0607ee56cbfdf606bab62cda7818deb" Jul 22 19:03:33.895873 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:33.895848 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" Jul 22 19:03:34.030542 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:34.030484 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/334f8642-4498-4c8c-9c13-d810152519fa-bundle\") pod \"334f8642-4498-4c8c-9c13-d810152519fa\" (UID: \"334f8642-4498-4c8c-9c13-d810152519fa\") " Jul 22 19:03:34.030724 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:34.030563 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/334f8642-4498-4c8c-9c13-d810152519fa-util\") pod \"334f8642-4498-4c8c-9c13-d810152519fa\" (UID: \"334f8642-4498-4c8c-9c13-d810152519fa\") " Jul 22 19:03:34.030724 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:34.030640 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfdwj\" (UniqueName: \"kubernetes.io/projected/334f8642-4498-4c8c-9c13-d810152519fa-kube-api-access-mfdwj\") pod \"334f8642-4498-4c8c-9c13-d810152519fa\" (UID: \"334f8642-4498-4c8c-9c13-d810152519fa\") " Jul 22 19:03:34.031125 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:34.031082 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334f8642-4498-4c8c-9c13-d810152519fa-bundle" (OuterVolumeSpecName: "bundle") pod "334f8642-4498-4c8c-9c13-d810152519fa" (UID: "334f8642-4498-4c8c-9c13-d810152519fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:03:34.032978 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:34.032945 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334f8642-4498-4c8c-9c13-d810152519fa-kube-api-access-mfdwj" (OuterVolumeSpecName: "kube-api-access-mfdwj") pod "334f8642-4498-4c8c-9c13-d810152519fa" (UID: "334f8642-4498-4c8c-9c13-d810152519fa"). InnerVolumeSpecName "kube-api-access-mfdwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:03:34.036212 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:34.036187 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334f8642-4498-4c8c-9c13-d810152519fa-util" (OuterVolumeSpecName: "util") pod "334f8642-4498-4c8c-9c13-d810152519fa" (UID: "334f8642-4498-4c8c-9c13-d810152519fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:03:34.132172 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:34.132083 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfdwj\" (UniqueName: \"kubernetes.io/projected/334f8642-4498-4c8c-9c13-d810152519fa-kube-api-access-mfdwj\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:03:34.132172 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:34.132113 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/334f8642-4498-4c8c-9c13-d810152519fa-bundle\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:03:34.132172 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:34.132124 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/334f8642-4498-4c8c-9c13-d810152519fa-util\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:03:34.758130 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:34.758100 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" event={"ID":"334f8642-4498-4c8c-9c13-d810152519fa","Type":"ContainerDied","Data":"0c3943085be8035a470c596ac9568548ea803d39f582bcc989d1472485801d69"} Jul 22 19:03:34.758130 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:34.758132 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq" Jul 22 19:03:34.758594 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:34.758150 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c3943085be8035a470c596ac9568548ea803d39f582bcc989d1472485801d69" Jul 22 19:03:38.278075 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278029 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-598df6b7d6-zg2jr"] Jul 22 19:03:38.278475 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278459 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd614366-354d-4198-a00b-c4bad2d27b92" containerName="extract" Jul 22 19:03:38.278549 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278477 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd614366-354d-4198-a00b-c4bad2d27b92" containerName="extract" Jul 22 19:03:38.278549 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278486 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="334f8642-4498-4c8c-9c13-d810152519fa" containerName="pull" Jul 22 19:03:38.278549 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278508 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="334f8642-4498-4c8c-9c13-d810152519fa" containerName="pull" Jul 22 19:03:38.278549 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278516 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6371b6ea-2108-4ab2-8d67-8ded5127613a" containerName="pull" Jul 22 19:03:38.278549 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278521 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6371b6ea-2108-4ab2-8d67-8ded5127613a" containerName="pull" Jul 22 19:03:38.278549 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278528 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6371b6ea-2108-4ab2-8d67-8ded5127613a" containerName="util" Jul 22 19:03:38.278549 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278533 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6371b6ea-2108-4ab2-8d67-8ded5127613a" containerName="util" Jul 22 19:03:38.278549 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278542 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd614366-354d-4198-a00b-c4bad2d27b92" containerName="pull" Jul 22 19:03:38.278549 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278547 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd614366-354d-4198-a00b-c4bad2d27b92" containerName="pull" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278554 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc860783-da97-4027-bce5-75d91c7dd87d" containerName="util" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278560 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc860783-da97-4027-bce5-75d91c7dd87d" containerName="util" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278570 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc860783-da97-4027-bce5-75d91c7dd87d" containerName="extract" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278574 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc860783-da97-4027-bce5-75d91c7dd87d" containerName="extract" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278580 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6371b6ea-2108-4ab2-8d67-8ded5127613a" containerName="extract" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278585 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6371b6ea-2108-4ab2-8d67-8ded5127613a" containerName="extract" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278594 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd614366-354d-4198-a00b-c4bad2d27b92" containerName="util" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278599 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd614366-354d-4198-a00b-c4bad2d27b92" containerName="util" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278604 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="334f8642-4498-4c8c-9c13-d810152519fa" containerName="util" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278609 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="334f8642-4498-4c8c-9c13-d810152519fa" containerName="util" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278617 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="334f8642-4498-4c8c-9c13-d810152519fa" containerName="extract" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278622 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="334f8642-4498-4c8c-9c13-d810152519fa" containerName="extract" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278630 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc860783-da97-4027-bce5-75d91c7dd87d" containerName="pull" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278636 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc860783-da97-4027-bce5-75d91c7dd87d" containerName="pull" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278694 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc860783-da97-4027-bce5-75d91c7dd87d" containerName="extract" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278704 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="334f8642-4498-4c8c-9c13-d810152519fa" containerName="extract" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278712 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6371b6ea-2108-4ab2-8d67-8ded5127613a" containerName="extract" Jul 22 19:03:38.278822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.278717 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd614366-354d-4198-a00b-c4bad2d27b92" containerName="extract" Jul 22 19:03:38.282257 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.282231 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.296024 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.295995 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-598df6b7d6-zg2jr"] Jul 22 19:03:38.371637 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.371595 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18389c84-6ee5-46c8-a5aa-a6caac93d353-service-ca\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.371637 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.371637 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18389c84-6ee5-46c8-a5aa-a6caac93d353-console-config\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.371873 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.371668 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18389c84-6ee5-46c8-a5aa-a6caac93d353-trusted-ca-bundle\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.371873 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.371789 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18389c84-6ee5-46c8-a5aa-a6caac93d353-console-oauth-config\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.371873 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.371839 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18389c84-6ee5-46c8-a5aa-a6caac93d353-oauth-serving-cert\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.371873 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.371863 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18389c84-6ee5-46c8-a5aa-a6caac93d353-console-serving-cert\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.372022 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.371879 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf22w\" (UniqueName: \"kubernetes.io/projected/18389c84-6ee5-46c8-a5aa-a6caac93d353-kube-api-access-kf22w\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.473174 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.473132 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18389c84-6ee5-46c8-a5aa-a6caac93d353-service-ca\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.473174 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.473177 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18389c84-6ee5-46c8-a5aa-a6caac93d353-console-config\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.473443 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.473204 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18389c84-6ee5-46c8-a5aa-a6caac93d353-trusted-ca-bundle\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.473443 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.473267 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18389c84-6ee5-46c8-a5aa-a6caac93d353-console-oauth-config\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.473443 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.473284 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18389c84-6ee5-46c8-a5aa-a6caac93d353-oauth-serving-cert\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.473443 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.473299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18389c84-6ee5-46c8-a5aa-a6caac93d353-console-serving-cert\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.473443 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.473351 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf22w\" (UniqueName: \"kubernetes.io/projected/18389c84-6ee5-46c8-a5aa-a6caac93d353-kube-api-access-kf22w\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.474001 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.473974 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18389c84-6ee5-46c8-a5aa-a6caac93d353-console-config\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.474162 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.474072 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18389c84-6ee5-46c8-a5aa-a6caac93d353-service-ca\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.474211 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.474182 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18389c84-6ee5-46c8-a5aa-a6caac93d353-trusted-ca-bundle\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.474273 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.474120 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18389c84-6ee5-46c8-a5aa-a6caac93d353-oauth-serving-cert\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.475910 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.475884 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18389c84-6ee5-46c8-a5aa-a6caac93d353-console-oauth-config\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.476075 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.476055 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18389c84-6ee5-46c8-a5aa-a6caac93d353-console-serving-cert\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.481527 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.481488 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf22w\" (UniqueName: \"kubernetes.io/projected/18389c84-6ee5-46c8-a5aa-a6caac93d353-kube-api-access-kf22w\") pod \"console-598df6b7d6-zg2jr\" (UID: \"18389c84-6ee5-46c8-a5aa-a6caac93d353\") " pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.593286 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.593185 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:38.735063 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.735031 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-598df6b7d6-zg2jr"] Jul 22 19:03:38.737027 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:03:38.737001 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18389c84_6ee5_46c8_a5aa_a6caac93d353.slice/crio-5babf20213cc4593718c35a7eab8578e27c11b1e00bc17f7d293fc7afc93f2c3 WatchSource:0}: Error finding container 5babf20213cc4593718c35a7eab8578e27c11b1e00bc17f7d293fc7afc93f2c3: Status 404 returned error can't find the container with id 5babf20213cc4593718c35a7eab8578e27c11b1e00bc17f7d293fc7afc93f2c3 Jul 22 19:03:38.783785 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:38.783753 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-598df6b7d6-zg2jr" event={"ID":"18389c84-6ee5-46c8-a5aa-a6caac93d353","Type":"ContainerStarted","Data":"5babf20213cc4593718c35a7eab8578e27c11b1e00bc17f7d293fc7afc93f2c3"} Jul 22 19:03:39.791458 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:39.791421 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-598df6b7d6-zg2jr" event={"ID":"18389c84-6ee5-46c8-a5aa-a6caac93d353","Type":"ContainerStarted","Data":"1c2710b4978c50ed112cb0113508fa07016223a8448735355ce07ee1e2a003cd"} Jul 22 19:03:39.811613 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:39.811559 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-598df6b7d6-zg2jr" podStartSLOduration=1.8115443070000001 podStartE2EDuration="1.811544307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:03:39.809539314 +0000 UTC m=+422.721703383" watchObservedRunningTime="2026-07-22 19:03:39.811544307 +0000 UTC m=+422.723708416" Jul 22 19:03:48.593717 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:48.593672 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:48.594139 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:48.593762 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:48.598553 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:48.598527 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:48.849800 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:48.849711 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-598df6b7d6-zg2jr" Jul 22 19:03:48.979308 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:48.979267 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86c97f59d-x9c5s"] Jul 22 19:03:52.477526 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:52.477474 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm"] Jul 22 19:03:52.482176 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:52.482158 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm" Jul 22 19:03:52.485725 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:52.485705 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 19:03:52.486916 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:52.486897 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5xk9k\"" Jul 22 19:03:52.487016 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:52.486903 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 19:03:52.501793 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:52.501766 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dxp7\" (UniqueName: \"kubernetes.io/projected/f06e573c-dfb0-4a02-b79b-25e94be009c8-kube-api-access-5dxp7\") pod \"limitador-operator-controller-manager-859d7b78bd-5lqbm\" (UID: \"f06e573c-dfb0-4a02-b79b-25e94be009c8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm" Jul 22 19:03:52.528081 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:52.528049 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm"] Jul 22 19:03:52.602477 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:52.602438 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dxp7\" (UniqueName: \"kubernetes.io/projected/f06e573c-dfb0-4a02-b79b-25e94be009c8-kube-api-access-5dxp7\") pod \"limitador-operator-controller-manager-859d7b78bd-5lqbm\" (UID: \"f06e573c-dfb0-4a02-b79b-25e94be009c8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm" Jul 22 19:03:52.611926 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:52.611898 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dxp7\" (UniqueName: \"kubernetes.io/projected/f06e573c-dfb0-4a02-b79b-25e94be009c8-kube-api-access-5dxp7\") pod \"limitador-operator-controller-manager-859d7b78bd-5lqbm\" (UID: \"f06e573c-dfb0-4a02-b79b-25e94be009c8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm" Jul 22 19:03:52.792671 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:52.792630 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm" Jul 22 19:03:52.942566 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:52.942539 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm"] Jul 22 19:03:52.943394 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:03:52.943361 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06e573c_dfb0_4a02_b79b_25e94be009c8.slice/crio-9ff6b8acb3caae6820e4485b37099685002355aa422680d4b1a423c75f4d2ac1 WatchSource:0}: Error finding container 9ff6b8acb3caae6820e4485b37099685002355aa422680d4b1a423c75f4d2ac1: Status 404 returned error can't find the container with id 9ff6b8acb3caae6820e4485b37099685002355aa422680d4b1a423c75f4d2ac1 Jul 22 19:03:53.882163 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:53.882117 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm" event={"ID":"f06e573c-dfb0-4a02-b79b-25e94be009c8","Type":"ContainerStarted","Data":"9ff6b8acb3caae6820e4485b37099685002355aa422680d4b1a423c75f4d2ac1"} Jul 22 19:03:54.889365 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:54.889325 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm" event={"ID":"f06e573c-dfb0-4a02-b79b-25e94be009c8","Type":"ContainerStarted","Data":"bc7ef71599a255aeeb81e66b591f6e5161ce003483d1119930e223dd91634456"} Jul 22 19:03:54.889786 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:54.889402 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm" Jul 22 19:03:54.910080 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:03:54.910024 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm" podStartSLOduration=1.063234881 podStartE2EDuration="2.910006801s" totalImagesPullingTime="1.84677192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:03:54.907456551 +0000 UTC m=+437.819620618" watchObservedRunningTime="2026-07-22 19:03:54.910006801 +0000 UTC m=+437.822170868" Jul 22 19:04:05.898212 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:05.898128 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm" Jul 22 19:04:07.111055 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.111018 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck"] Jul 22 19:04:07.115834 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.115815 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck" Jul 22 19:04:07.118452 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.118432 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mwjz4\"" Jul 22 19:04:07.128148 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.128121 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck"] Jul 22 19:04:07.131872 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.131848 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phmj2\" (UniqueName: \"kubernetes.io/projected/14359555-ff66-4ac4-a156-121a977f4b28-kube-api-access-phmj2\") pod \"kuadrant-operator-controller-manager-55467d9f7-998ck\" (UID: \"14359555-ff66-4ac4-a156-121a977f4b28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck" Jul 22 19:04:07.131984 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.131930 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14359555-ff66-4ac4-a156-121a977f4b28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-998ck\" (UID: \"14359555-ff66-4ac4-a156-121a977f4b28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck" Jul 22 19:04:07.222109 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.222072 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck"] Jul 22 19:04:07.222357 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:04:07.222337 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-phmj2], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck" podUID="14359555-ff66-4ac4-a156-121a977f4b28" Jul 22 19:04:07.232789 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.232750 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14359555-ff66-4ac4-a156-121a977f4b28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-998ck\" (UID: \"14359555-ff66-4ac4-a156-121a977f4b28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck" Jul 22 19:04:07.232958 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.232840 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phmj2\" (UniqueName: \"kubernetes.io/projected/14359555-ff66-4ac4-a156-121a977f4b28-kube-api-access-phmj2\") pod \"kuadrant-operator-controller-manager-55467d9f7-998ck\" (UID: \"14359555-ff66-4ac4-a156-121a977f4b28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck" Jul 22 19:04:07.233151 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.233129 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14359555-ff66-4ac4-a156-121a977f4b28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-998ck\" (UID: \"14359555-ff66-4ac4-a156-121a977f4b28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck" Jul 22 19:04:07.246408 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.246368 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phmj2\" (UniqueName: \"kubernetes.io/projected/14359555-ff66-4ac4-a156-121a977f4b28-kube-api-access-phmj2\") pod \"kuadrant-operator-controller-manager-55467d9f7-998ck\" (UID: \"14359555-ff66-4ac4-a156-121a977f4b28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck" Jul 22 19:04:07.299985 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.299949 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck"] Jul 22 19:04:07.302532 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.302480 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck"] Jul 22 19:04:07.386682 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.386581 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm"] Jul 22 19:04:07.386843 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.386781 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm" podUID="f06e573c-dfb0-4a02-b79b-25e94be009c8" containerName="manager" containerID="cri-o://bc7ef71599a255aeeb81e66b591f6e5161ce003483d1119930e223dd91634456" gracePeriod=2 Jul 22 19:04:07.394735 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.394695 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm"] Jul 22 19:04:07.400813 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:04:07.400732 2583 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"f06e573c-dfb0-4a02-b79b-25e94be009c8\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-5lqbm\": pods \"limitador-operator-controller-manager-859d7b78bd-5lqbm\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm" Jul 22 19:04:07.415717 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.415679 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hkkt6"] Jul 22 19:04:07.416138 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.416123 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f06e573c-dfb0-4a02-b79b-25e94be009c8" containerName="manager" Jul 22 19:04:07.416182 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.416142 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f06e573c-dfb0-4a02-b79b-25e94be009c8" containerName="manager" Jul 22 19:04:07.416221 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.416213 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f06e573c-dfb0-4a02-b79b-25e94be009c8" containerName="manager" Jul 22 19:04:07.419136 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.419114 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hkkt6" Jul 22 19:04:07.421625 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:04:07.421587 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5lqbm\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="f06e573c-dfb0-4a02-b79b-25e94be009c8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm" Jul 22 19:04:07.429842 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.429719 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hkkt6"] Jul 22 19:04:07.435151 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.435117 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlfcq\" (UniqueName: \"kubernetes.io/projected/4d1f34a2-bf52-4511-9596-d1aaef7f0031-kube-api-access-hlfcq\") pod \"limitador-operator-controller-manager-859d7b78bd-hkkt6\" (UID: \"4d1f34a2-bf52-4511-9596-d1aaef7f0031\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hkkt6" Jul 22 19:04:07.536350 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.536316 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlfcq\" (UniqueName: \"kubernetes.io/projected/4d1f34a2-bf52-4511-9596-d1aaef7f0031-kube-api-access-hlfcq\") pod \"limitador-operator-controller-manager-859d7b78bd-hkkt6\" (UID: \"4d1f34a2-bf52-4511-9596-d1aaef7f0031\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hkkt6" Jul 22 19:04:07.546395 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.546361 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlfcq\" (UniqueName: \"kubernetes.io/projected/4d1f34a2-bf52-4511-9596-d1aaef7f0031-kube-api-access-hlfcq\") pod \"limitador-operator-controller-manager-859d7b78bd-hkkt6\" (UID: \"4d1f34a2-bf52-4511-9596-d1aaef7f0031\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hkkt6" Jul 22 19:04:07.550027 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:04:07.549986 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5lqbm\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="f06e573c-dfb0-4a02-b79b-25e94be009c8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm" Jul 22 19:04:07.630602 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.630574 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm" Jul 22 19:04:07.737599 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.737517 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dxp7\" (UniqueName: \"kubernetes.io/projected/f06e573c-dfb0-4a02-b79b-25e94be009c8-kube-api-access-5dxp7\") pod \"f06e573c-dfb0-4a02-b79b-25e94be009c8\" (UID: \"f06e573c-dfb0-4a02-b79b-25e94be009c8\") " Jul 22 19:04:07.739628 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.739595 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06e573c-dfb0-4a02-b79b-25e94be009c8-kube-api-access-5dxp7" (OuterVolumeSpecName: "kube-api-access-5dxp7") pod "f06e573c-dfb0-4a02-b79b-25e94be009c8" (UID: "f06e573c-dfb0-4a02-b79b-25e94be009c8"). InnerVolumeSpecName "kube-api-access-5dxp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:04:07.773737 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.773689 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hkkt6" Jul 22 19:04:07.838907 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.838864 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dxp7\" (UniqueName: \"kubernetes.io/projected/f06e573c-dfb0-4a02-b79b-25e94be009c8-kube-api-access-5dxp7\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:04:07.916137 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.916109 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hkkt6"] Jul 22 19:04:07.917411 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:04:07.917386 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1f34a2_bf52_4511_9596_d1aaef7f0031.slice/crio-adcba6f2541a3e3cfdcc75cedd1a71de176fd7be9898e6999320568eaf8fb493 WatchSource:0}: Error finding container adcba6f2541a3e3cfdcc75cedd1a71de176fd7be9898e6999320568eaf8fb493: Status 404 returned error can't find the container with id adcba6f2541a3e3cfdcc75cedd1a71de176fd7be9898e6999320568eaf8fb493 Jul 22 19:04:07.964243 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.964205 2583 generic.go:356] "Generic (PLEG): container finished" podID="f06e573c-dfb0-4a02-b79b-25e94be009c8" containerID="bc7ef71599a255aeeb81e66b591f6e5161ce003483d1119930e223dd91634456" exitCode=0 Jul 22 19:04:07.964408 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.964265 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5lqbm" Jul 22 19:04:07.964877 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.964297 2583 scope.go:117] "RemoveContainer" containerID="bc7ef71599a255aeeb81e66b591f6e5161ce003483d1119930e223dd91634456" Jul 22 19:04:07.965655 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.965609 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hkkt6" event={"ID":"4d1f34a2-bf52-4511-9596-d1aaef7f0031","Type":"ContainerStarted","Data":"adcba6f2541a3e3cfdcc75cedd1a71de176fd7be9898e6999320568eaf8fb493"} Jul 22 19:04:07.965655 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.965649 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck" Jul 22 19:04:07.975311 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.975289 2583 scope.go:117] "RemoveContainer" containerID="bc7ef71599a255aeeb81e66b591f6e5161ce003483d1119930e223dd91634456" Jul 22 19:04:07.975587 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.975570 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck" Jul 22 19:04:07.975653 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:04:07.975626 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7ef71599a255aeeb81e66b591f6e5161ce003483d1119930e223dd91634456\": container with ID starting with bc7ef71599a255aeeb81e66b591f6e5161ce003483d1119930e223dd91634456 not found: ID does not exist" containerID="bc7ef71599a255aeeb81e66b591f6e5161ce003483d1119930e223dd91634456" Jul 22 19:04:07.975691 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:07.975655 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7ef71599a255aeeb81e66b591f6e5161ce003483d1119930e223dd91634456"} err="failed to get container status \"bc7ef71599a255aeeb81e66b591f6e5161ce003483d1119930e223dd91634456\": rpc error: code = NotFound desc = could not find container \"bc7ef71599a255aeeb81e66b591f6e5161ce003483d1119930e223dd91634456\": container with ID starting with bc7ef71599a255aeeb81e66b591f6e5161ce003483d1119930e223dd91634456 not found: ID does not exist" Jul 22 19:04:07.978393 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:04:07.978370 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-998ck\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="14359555-ff66-4ac4-a156-121a977f4b28" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck" Jul 22 19:04:08.040583 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:08.040549 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14359555-ff66-4ac4-a156-121a977f4b28-extensions-socket-volume\") pod \"14359555-ff66-4ac4-a156-121a977f4b28\" (UID: \"14359555-ff66-4ac4-a156-121a977f4b28\") " Jul 22 19:04:08.040769 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:08.040643 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phmj2\" (UniqueName: \"kubernetes.io/projected/14359555-ff66-4ac4-a156-121a977f4b28-kube-api-access-phmj2\") pod \"14359555-ff66-4ac4-a156-121a977f4b28\" (UID: \"14359555-ff66-4ac4-a156-121a977f4b28\") " Jul 22 19:04:08.040826 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:08.040801 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14359555-ff66-4ac4-a156-121a977f4b28-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "14359555-ff66-4ac4-a156-121a977f4b28" (UID: "14359555-ff66-4ac4-a156-121a977f4b28"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 19:04:08.042760 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:08.042733 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14359555-ff66-4ac4-a156-121a977f4b28-kube-api-access-phmj2" (OuterVolumeSpecName: "kube-api-access-phmj2") pod "14359555-ff66-4ac4-a156-121a977f4b28" (UID: "14359555-ff66-4ac4-a156-121a977f4b28"). InnerVolumeSpecName "kube-api-access-phmj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:04:08.141881 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:08.141839 2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14359555-ff66-4ac4-a156-121a977f4b28-extensions-socket-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:04:08.141881 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:08.141873 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phmj2\" (UniqueName: \"kubernetes.io/projected/14359555-ff66-4ac4-a156-121a977f4b28-kube-api-access-phmj2\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:04:08.974210 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:08.974183 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck" Jul 22 19:04:08.974431 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:08.974228 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hkkt6" event={"ID":"4d1f34a2-bf52-4511-9596-d1aaef7f0031","Type":"ContainerStarted","Data":"f1acbf5b3822a2ca9ac8b568a5e892fea3e51c66c6a09cf8b5ea667ce7fa0252"} Jul 22 19:04:08.974431 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:08.974362 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hkkt6" Jul 22 19:04:08.976621 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:04:08.976587 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-998ck\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="14359555-ff66-4ac4-a156-121a977f4b28" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck" Jul 22 19:04:08.997846 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:04:08.997808 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-998ck\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="14359555-ff66-4ac4-a156-121a977f4b28" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-998ck" Jul 22 19:04:08.998673 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:08.998633 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hkkt6" podStartSLOduration=1.998618373 podStartE2EDuration="1.998618373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:04:08.995622295 +0000 UTC m=+451.907786363" watchObservedRunningTime="2026-07-22 19:04:08.998618373 +0000 UTC m=+451.910782441" Jul 22 19:04:09.550258 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:09.550222 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14359555-ff66-4ac4-a156-121a977f4b28" path="/var/lib/kubelet/pods/14359555-ff66-4ac4-a156-121a977f4b28/volumes" Jul 22 19:04:09.550656 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:09.550453 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f06e573c-dfb0-4a02-b79b-25e94be009c8" path="/var/lib/kubelet/pods/f06e573c-dfb0-4a02-b79b-25e94be009c8/volumes" Jul 22 19:04:14.006858 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.006799 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86c97f59d-x9c5s" podUID="81e25488-346a-45b5-8b9a-350144d69c05" containerName="console" containerID="cri-o://c210e13d094777720d1474078be25b5ecf383bc50e521a210cd8e039d48dbe36" gracePeriod=15 Jul 22 19:04:14.267021 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.266999 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 19:04:14.295569 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.295535 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-console-config\") pod \"81e25488-346a-45b5-8b9a-350144d69c05\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " Jul 22 19:04:14.295735 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.295632 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-service-ca\") pod \"81e25488-346a-45b5-8b9a-350144d69c05\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " Jul 22 19:04:14.295735 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.295665 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94pc8\" (UniqueName: \"kubernetes.io/projected/81e25488-346a-45b5-8b9a-350144d69c05-kube-api-access-94pc8\") pod \"81e25488-346a-45b5-8b9a-350144d69c05\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " Jul 22 19:04:14.295735 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.295684 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81e25488-346a-45b5-8b9a-350144d69c05-console-oauth-config\") pod \"81e25488-346a-45b5-8b9a-350144d69c05\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " Jul 22 19:04:14.295735 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.295729 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81e25488-346a-45b5-8b9a-350144d69c05-console-serving-cert\") pod \"81e25488-346a-45b5-8b9a-350144d69c05\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " Jul 22 19:04:14.295950 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.295765 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-trusted-ca-bundle\") pod \"81e25488-346a-45b5-8b9a-350144d69c05\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " Jul 22 19:04:14.295950 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.295795 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-oauth-serving-cert\") pod \"81e25488-346a-45b5-8b9a-350144d69c05\" (UID: \"81e25488-346a-45b5-8b9a-350144d69c05\") " Jul 22 19:04:14.295950 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.295932 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-console-config" (OuterVolumeSpecName: "console-config") pod "81e25488-346a-45b5-8b9a-350144d69c05" (UID: "81e25488-346a-45b5-8b9a-350144d69c05"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:04:14.296113 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.296067 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-service-ca" (OuterVolumeSpecName: "service-ca") pod "81e25488-346a-45b5-8b9a-350144d69c05" (UID: "81e25488-346a-45b5-8b9a-350144d69c05"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:04:14.296245 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.296225 2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-console-config\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:04:14.296309 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.296249 2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:04:14.296368 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.296305 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "81e25488-346a-45b5-8b9a-350144d69c05" (UID: "81e25488-346a-45b5-8b9a-350144d69c05"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:04:14.296525 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.296487 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "81e25488-346a-45b5-8b9a-350144d69c05" (UID: "81e25488-346a-45b5-8b9a-350144d69c05"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:04:14.298531 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.298449 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e25488-346a-45b5-8b9a-350144d69c05-kube-api-access-94pc8" (OuterVolumeSpecName: "kube-api-access-94pc8") pod "81e25488-346a-45b5-8b9a-350144d69c05" (UID: "81e25488-346a-45b5-8b9a-350144d69c05"). InnerVolumeSpecName "kube-api-access-94pc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:04:14.298648 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.298545 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81e25488-346a-45b5-8b9a-350144d69c05-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "81e25488-346a-45b5-8b9a-350144d69c05" (UID: "81e25488-346a-45b5-8b9a-350144d69c05"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:04:14.298648 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.298561 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81e25488-346a-45b5-8b9a-350144d69c05-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "81e25488-346a-45b5-8b9a-350144d69c05" (UID: "81e25488-346a-45b5-8b9a-350144d69c05"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:04:14.396860 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.396819 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94pc8\" (UniqueName: \"kubernetes.io/projected/81e25488-346a-45b5-8b9a-350144d69c05-kube-api-access-94pc8\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:04:14.396860 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.396861 2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81e25488-346a-45b5-8b9a-350144d69c05-console-oauth-config\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:04:14.397057 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.396877 2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81e25488-346a-45b5-8b9a-350144d69c05-console-serving-cert\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:04:14.397057 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.396891 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-trusted-ca-bundle\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:04:14.397057 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:14.396907 2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81e25488-346a-45b5-8b9a-350144d69c05-oauth-serving-cert\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:04:15.010645 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:15.010612 2583 generic.go:356] "Generic (PLEG): container finished" podID="81e25488-346a-45b5-8b9a-350144d69c05" containerID="c210e13d094777720d1474078be25b5ecf383bc50e521a210cd8e039d48dbe36" exitCode=0 Jul 22 19:04:15.011071 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:15.010654 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86c97f59d-x9c5s" Jul 22 19:04:15.011071 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:15.010696 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86c97f59d-x9c5s" event={"ID":"81e25488-346a-45b5-8b9a-350144d69c05","Type":"ContainerDied","Data":"c210e13d094777720d1474078be25b5ecf383bc50e521a210cd8e039d48dbe36"} Jul 22 19:04:15.011071 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:15.010748 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86c97f59d-x9c5s" event={"ID":"81e25488-346a-45b5-8b9a-350144d69c05","Type":"ContainerDied","Data":"01a06c423b6d00c555169ec3a7a59a7eb8520d3cbe52b03d1cd8bc127fdf96d1"} Jul 22 19:04:15.011071 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:15.010769 2583 scope.go:117] "RemoveContainer" containerID="c210e13d094777720d1474078be25b5ecf383bc50e521a210cd8e039d48dbe36" Jul 22 19:04:15.020677 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:15.020658 2583 scope.go:117] "RemoveContainer" containerID="c210e13d094777720d1474078be25b5ecf383bc50e521a210cd8e039d48dbe36" Jul 22 19:04:15.020980 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:04:15.020961 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c210e13d094777720d1474078be25b5ecf383bc50e521a210cd8e039d48dbe36\": container with ID starting with c210e13d094777720d1474078be25b5ecf383bc50e521a210cd8e039d48dbe36 not found: ID does not exist" containerID="c210e13d094777720d1474078be25b5ecf383bc50e521a210cd8e039d48dbe36" Jul 22 19:04:15.021019 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:15.020992 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c210e13d094777720d1474078be25b5ecf383bc50e521a210cd8e039d48dbe36"} err="failed to get container status \"c210e13d094777720d1474078be25b5ecf383bc50e521a210cd8e039d48dbe36\": rpc error: code = NotFound desc = could not find container \"c210e13d094777720d1474078be25b5ecf383bc50e521a210cd8e039d48dbe36\": container with ID starting with c210e13d094777720d1474078be25b5ecf383bc50e521a210cd8e039d48dbe36 not found: ID does not exist" Jul 22 19:04:15.044690 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:15.044653 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86c97f59d-x9c5s"] Jul 22 19:04:15.054360 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:15.054329 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86c97f59d-x9c5s"] Jul 22 19:04:15.550773 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:15.550744 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81e25488-346a-45b5-8b9a-350144d69c05" path="/var/lib/kubelet/pods/81e25488-346a-45b5-8b9a-350144d69c05/volumes" Jul 22 19:04:19.983922 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:19.983891 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hkkt6" Jul 22 19:04:54.950619 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:54.950581 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mnd2d"] Jul 22 19:04:54.951043 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:54.951008 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81e25488-346a-45b5-8b9a-350144d69c05" containerName="console" Jul 22 19:04:54.951043 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:54.951020 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="81e25488-346a-45b5-8b9a-350144d69c05" containerName="console" Jul 22 19:04:54.951125 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:54.951080 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="81e25488-346a-45b5-8b9a-350144d69c05" containerName="console" Jul 22 19:04:54.955730 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:54.955713 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" Jul 22 19:04:54.958583 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:54.958555 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 19:04:54.958701 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:54.958564 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7mkgk\"" Jul 22 19:04:54.963799 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:54.963755 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mnd2d"] Jul 22 19:04:54.970698 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:54.970642 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc2c4007-4586-4142-a734-d81376b750e3-config-file\") pod \"limitador-limitador-64c94fd64f-mnd2d\" (UID: \"bc2c4007-4586-4142-a734-d81376b750e3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" Jul 22 19:04:54.970856 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:54.970714 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrpk7\" (UniqueName: \"kubernetes.io/projected/bc2c4007-4586-4142-a734-d81376b750e3-kube-api-access-vrpk7\") pod \"limitador-limitador-64c94fd64f-mnd2d\" (UID: \"bc2c4007-4586-4142-a734-d81376b750e3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" Jul 22 19:04:55.053476 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:55.053443 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mnd2d"] Jul 22 19:04:55.072023 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:55.071992 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc2c4007-4586-4142-a734-d81376b750e3-config-file\") pod \"limitador-limitador-64c94fd64f-mnd2d\" (UID: \"bc2c4007-4586-4142-a734-d81376b750e3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" Jul 22 19:04:55.072181 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:55.072031 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrpk7\" (UniqueName: \"kubernetes.io/projected/bc2c4007-4586-4142-a734-d81376b750e3-kube-api-access-vrpk7\") pod \"limitador-limitador-64c94fd64f-mnd2d\" (UID: \"bc2c4007-4586-4142-a734-d81376b750e3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" Jul 22 19:04:55.072643 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:55.072623 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc2c4007-4586-4142-a734-d81376b750e3-config-file\") pod \"limitador-limitador-64c94fd64f-mnd2d\" (UID: \"bc2c4007-4586-4142-a734-d81376b750e3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" Jul 22 19:04:55.080763 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:55.080737 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrpk7\" (UniqueName: \"kubernetes.io/projected/bc2c4007-4586-4142-a734-d81376b750e3-kube-api-access-vrpk7\") pod \"limitador-limitador-64c94fd64f-mnd2d\" (UID: \"bc2c4007-4586-4142-a734-d81376b750e3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" Jul 22 19:04:55.267744 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:55.267720 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" Jul 22 19:04:55.410407 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:55.410373 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mnd2d"] Jul 22 19:04:55.412105 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:04:55.412079 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2c4007_4586_4142_a734_d81376b750e3.slice/crio-2a993e7ae9c1e3db46a01bde096cb89300753fce2fa11bea9d60592187971756 WatchSource:0}: Error finding container 2a993e7ae9c1e3db46a01bde096cb89300753fce2fa11bea9d60592187971756: Status 404 returned error can't find the container with id 2a993e7ae9c1e3db46a01bde096cb89300753fce2fa11bea9d60592187971756 Jul 22 19:04:56.249989 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:56.249899 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" event={"ID":"bc2c4007-4586-4142-a734-d81376b750e3","Type":"ContainerStarted","Data":"2a993e7ae9c1e3db46a01bde096cb89300753fce2fa11bea9d60592187971756"} Jul 22 19:04:58.264119 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:58.264078 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" event={"ID":"bc2c4007-4586-4142-a734-d81376b750e3","Type":"ContainerStarted","Data":"4909b105fb72b6e55e33848eefd172f86ea7493e563283d44a84da1b751aa90d"} Jul 22 19:04:58.264547 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:58.264149 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" Jul 22 19:04:58.283618 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:04:58.283565 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" podStartSLOduration=1.586461498 podStartE2EDuration="4.283547207s" totalImagesPullingTime="2.697085709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:04:58.280985052 +0000 UTC m=+501.193149144" watchObservedRunningTime="2026-07-22 19:04:58.283547207 +0000 UTC m=+501.195711276" Jul 22 19:05:09.279089 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:09.279044 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" Jul 22 19:05:09.773934 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:09.773899 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mnd2d"] Jul 22 19:05:09.774143 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:09.774114 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" podUID="bc2c4007-4586-4142-a734-d81376b750e3" containerName="limitador" containerID="cri-o://4909b105fb72b6e55e33848eefd172f86ea7493e563283d44a84da1b751aa90d" gracePeriod=30 Jul 22 19:05:10.341914 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:10.341823 2583 generic.go:356] "Generic (PLEG): container finished" podID="bc2c4007-4586-4142-a734-d81376b750e3" containerID="4909b105fb72b6e55e33848eefd172f86ea7493e563283d44a84da1b751aa90d" exitCode=0 Jul 22 19:05:10.341914 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:10.341859 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" event={"ID":"bc2c4007-4586-4142-a734-d81376b750e3","Type":"ContainerDied","Data":"4909b105fb72b6e55e33848eefd172f86ea7493e563283d44a84da1b751aa90d"} Jul 22 19:05:10.342276 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:10.341944 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" event={"ID":"bc2c4007-4586-4142-a734-d81376b750e3","Type":"ContainerDied","Data":"2a993e7ae9c1e3db46a01bde096cb89300753fce2fa11bea9d60592187971756"} Jul 22 19:05:10.342276 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:10.341971 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a993e7ae9c1e3db46a01bde096cb89300753fce2fa11bea9d60592187971756" Jul 22 19:05:10.344224 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:10.344209 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" Jul 22 19:05:10.416890 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:10.416857 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrpk7\" (UniqueName: \"kubernetes.io/projected/bc2c4007-4586-4142-a734-d81376b750e3-kube-api-access-vrpk7\") pod \"bc2c4007-4586-4142-a734-d81376b750e3\" (UID: \"bc2c4007-4586-4142-a734-d81376b750e3\") " Jul 22 19:05:10.417099 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:10.416918 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc2c4007-4586-4142-a734-d81376b750e3-config-file\") pod \"bc2c4007-4586-4142-a734-d81376b750e3\" (UID: \"bc2c4007-4586-4142-a734-d81376b750e3\") " Jul 22 19:05:10.417287 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:10.417260 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc2c4007-4586-4142-a734-d81376b750e3-config-file" (OuterVolumeSpecName: "config-file") pod "bc2c4007-4586-4142-a734-d81376b750e3" (UID: "bc2c4007-4586-4142-a734-d81376b750e3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:05:10.419139 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:10.419114 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2c4007-4586-4142-a734-d81376b750e3-kube-api-access-vrpk7" (OuterVolumeSpecName: "kube-api-access-vrpk7") pod "bc2c4007-4586-4142-a734-d81376b750e3" (UID: "bc2c4007-4586-4142-a734-d81376b750e3"). InnerVolumeSpecName "kube-api-access-vrpk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:05:10.518533 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:10.518478 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrpk7\" (UniqueName: \"kubernetes.io/projected/bc2c4007-4586-4142-a734-d81376b750e3-kube-api-access-vrpk7\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:05:10.518533 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:10.518532 2583 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc2c4007-4586-4142-a734-d81376b750e3-config-file\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:05:11.286539 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.286489 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8mb8z"] Jul 22 19:05:11.286920 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.286907 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc2c4007-4586-4142-a734-d81376b750e3" containerName="limitador" Jul 22 19:05:11.286964 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.286922 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc2c4007-4586-4142-a734-d81376b750e3" containerName="limitador" Jul 22 19:05:11.286999 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.286991 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc2c4007-4586-4142-a734-d81376b750e3" containerName="limitador" Jul 22 19:05:11.290167 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.290147 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mb8z" Jul 22 19:05:11.293589 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.293570 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 22 19:05:11.293719 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.293592 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-l9wnw\"" Jul 22 19:05:11.294678 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.294655 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 22 19:05:11.294798 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.294665 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 19:05:11.337421 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.337388 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8mb8z"] Jul 22 19:05:11.347222 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.347195 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnd2d" Jul 22 19:05:11.369643 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.369610 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mnd2d"] Jul 22 19:05:11.374385 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.374359 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mnd2d"] Jul 22 19:05:11.426374 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.426337 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5ef60f74-9c3b-4e44-8e2d-fbd91d5a06cb-data\") pod \"postgres-69fb4594cf-8mb8z\" (UID: \"5ef60f74-9c3b-4e44-8e2d-fbd91d5a06cb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mb8z" Jul 22 19:05:11.426555 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.426383 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn9lf\" (UniqueName: \"kubernetes.io/projected/5ef60f74-9c3b-4e44-8e2d-fbd91d5a06cb-kube-api-access-wn9lf\") pod \"postgres-69fb4594cf-8mb8z\" (UID: \"5ef60f74-9c3b-4e44-8e2d-fbd91d5a06cb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mb8z" Jul 22 19:05:11.527696 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.527650 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5ef60f74-9c3b-4e44-8e2d-fbd91d5a06cb-data\") pod \"postgres-69fb4594cf-8mb8z\" (UID: \"5ef60f74-9c3b-4e44-8e2d-fbd91d5a06cb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mb8z" Jul 22 19:05:11.527891 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.527711 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn9lf\" (UniqueName: \"kubernetes.io/projected/5ef60f74-9c3b-4e44-8e2d-fbd91d5a06cb-kube-api-access-wn9lf\") pod \"postgres-69fb4594cf-8mb8z\" (UID: \"5ef60f74-9c3b-4e44-8e2d-fbd91d5a06cb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mb8z" Jul 22 19:05:11.528089 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.528068 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5ef60f74-9c3b-4e44-8e2d-fbd91d5a06cb-data\") pod \"postgres-69fb4594cf-8mb8z\" (UID: \"5ef60f74-9c3b-4e44-8e2d-fbd91d5a06cb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mb8z" Jul 22 19:05:11.536648 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.536580 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn9lf\" (UniqueName: \"kubernetes.io/projected/5ef60f74-9c3b-4e44-8e2d-fbd91d5a06cb-kube-api-access-wn9lf\") pod \"postgres-69fb4594cf-8mb8z\" (UID: \"5ef60f74-9c3b-4e44-8e2d-fbd91d5a06cb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mb8z" Jul 22 19:05:11.552063 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.552028 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc2c4007-4586-4142-a734-d81376b750e3" path="/var/lib/kubelet/pods/bc2c4007-4586-4142-a734-d81376b750e3/volumes" Jul 22 19:05:11.600273 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.600225 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mb8z" Jul 22 19:05:11.743550 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:11.743524 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8mb8z"] Jul 22 19:05:11.745083 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:05:11.745054 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef60f74_9c3b_4e44_8e2d_fbd91d5a06cb.slice/crio-83ad44e025fedc6185c369a69d1a93fd5426b1693e709035aa32edc5866037cc WatchSource:0}: Error finding container 83ad44e025fedc6185c369a69d1a93fd5426b1693e709035aa32edc5866037cc: Status 404 returned error can't find the container with id 83ad44e025fedc6185c369a69d1a93fd5426b1693e709035aa32edc5866037cc Jul 22 19:05:12.353891 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:12.353859 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mb8z" event={"ID":"5ef60f74-9c3b-4e44-8e2d-fbd91d5a06cb","Type":"ContainerStarted","Data":"83ad44e025fedc6185c369a69d1a93fd5426b1693e709035aa32edc5866037cc"} Jul 22 19:05:17.393671 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:17.393637 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mb8z" event={"ID":"5ef60f74-9c3b-4e44-8e2d-fbd91d5a06cb","Type":"ContainerStarted","Data":"608105812e0ef58fdc247c51c12f9a1cc812d0aef7e05d6d6bdf4062f08e990c"} Jul 22 19:05:17.394076 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:17.393714 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mb8z" Jul 22 19:05:17.413247 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:17.413182 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mb8z" podStartSLOduration=1.469967612 podStartE2EDuration="6.413165375s" totalImagesPullingTime="4.943197763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:05:17.411016411 +0000 UTC m=+520.323180482" watchObservedRunningTime="2026-07-22 19:05:17.413165375 +0000 UTC m=+520.325329444" Jul 22 19:05:23.430243 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:23.430214 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mb8z" Jul 22 19:05:33.519133 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.519045 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66fdbfd797-h4qgm"] Jul 22 19:05:33.524945 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.524925 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fdbfd797-h4qgm" Jul 22 19:05:33.528799 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.528772 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 19:05:33.528799 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.528785 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 19:05:33.529887 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.529873 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-g2pff\"" Jul 22 19:05:33.539283 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.539245 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/905f4e50-cd2a-48cb-81ae-c5d257e11665-webhook-cert\") pod \"maas-controller-66fdbfd797-h4qgm\" (UID: \"905f4e50-cd2a-48cb-81ae-c5d257e11665\") " pod="opendatahub/maas-controller-66fdbfd797-h4qgm" Jul 22 19:05:33.539423 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.539360 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrdtg\" (UniqueName: \"kubernetes.io/projected/905f4e50-cd2a-48cb-81ae-c5d257e11665-kube-api-access-rrdtg\") pod \"maas-controller-66fdbfd797-h4qgm\" (UID: \"905f4e50-cd2a-48cb-81ae-c5d257e11665\") " pod="opendatahub/maas-controller-66fdbfd797-h4qgm" Jul 22 19:05:33.540164 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.540141 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66fdbfd797-h4qgm"] Jul 22 19:05:33.640351 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.640317 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/905f4e50-cd2a-48cb-81ae-c5d257e11665-webhook-cert\") pod \"maas-controller-66fdbfd797-h4qgm\" (UID: \"905f4e50-cd2a-48cb-81ae-c5d257e11665\") " pod="opendatahub/maas-controller-66fdbfd797-h4qgm" Jul 22 19:05:33.640602 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.640362 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdtg\" (UniqueName: \"kubernetes.io/projected/905f4e50-cd2a-48cb-81ae-c5d257e11665-kube-api-access-rrdtg\") pod \"maas-controller-66fdbfd797-h4qgm\" (UID: \"905f4e50-cd2a-48cb-81ae-c5d257e11665\") " pod="opendatahub/maas-controller-66fdbfd797-h4qgm" Jul 22 19:05:33.640602 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:05:33.640459 2583 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 22 19:05:33.640602 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:05:33.640553 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/905f4e50-cd2a-48cb-81ae-c5d257e11665-webhook-cert podName:905f4e50-cd2a-48cb-81ae-c5d257e11665 nodeName:}" failed. No retries permitted until 2026-07-22 19:05:34.140534027 +0000 UTC m=+537.052698081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/905f4e50-cd2a-48cb-81ae-c5d257e11665-webhook-cert") pod "maas-controller-66fdbfd797-h4qgm" (UID: "905f4e50-cd2a-48cb-81ae-c5d257e11665") : secret "maas-controller-webhook-cert" not found Jul 22 19:05:33.660473 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.660433 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrdtg\" (UniqueName: \"kubernetes.io/projected/905f4e50-cd2a-48cb-81ae-c5d257e11665-kube-api-access-rrdtg\") pod \"maas-controller-66fdbfd797-h4qgm\" (UID: \"905f4e50-cd2a-48cb-81ae-c5d257e11665\") " pod="opendatahub/maas-controller-66fdbfd797-h4qgm" Jul 22 19:05:33.705512 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.705468 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d64d87758-5ksrd"] Jul 22 19:05:33.708450 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.708425 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d64d87758-5ksrd" Jul 22 19:05:33.722084 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.722051 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d64d87758-5ksrd"] Jul 22 19:05:33.741095 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.741066 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7517a2fd-6dab-41c1-9637-56f0c101f9b7-webhook-cert\") pod \"maas-controller-6d64d87758-5ksrd\" (UID: \"7517a2fd-6dab-41c1-9637-56f0c101f9b7\") " pod="opendatahub/maas-controller-6d64d87758-5ksrd" Jul 22 19:05:33.741275 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.741113 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44b6h\" (UniqueName: \"kubernetes.io/projected/7517a2fd-6dab-41c1-9637-56f0c101f9b7-kube-api-access-44b6h\") pod \"maas-controller-6d64d87758-5ksrd\" (UID: \"7517a2fd-6dab-41c1-9637-56f0c101f9b7\") " pod="opendatahub/maas-controller-6d64d87758-5ksrd" Jul 22 19:05:33.842454 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.842360 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7517a2fd-6dab-41c1-9637-56f0c101f9b7-webhook-cert\") pod \"maas-controller-6d64d87758-5ksrd\" (UID: \"7517a2fd-6dab-41c1-9637-56f0c101f9b7\") " pod="opendatahub/maas-controller-6d64d87758-5ksrd" Jul 22 19:05:33.842454 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.842408 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44b6h\" (UniqueName: \"kubernetes.io/projected/7517a2fd-6dab-41c1-9637-56f0c101f9b7-kube-api-access-44b6h\") pod \"maas-controller-6d64d87758-5ksrd\" (UID: \"7517a2fd-6dab-41c1-9637-56f0c101f9b7\") " pod="opendatahub/maas-controller-6d64d87758-5ksrd" Jul 22 19:05:33.845005 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.844980 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7517a2fd-6dab-41c1-9637-56f0c101f9b7-webhook-cert\") pod \"maas-controller-6d64d87758-5ksrd\" (UID: \"7517a2fd-6dab-41c1-9637-56f0c101f9b7\") " pod="opendatahub/maas-controller-6d64d87758-5ksrd" Jul 22 19:05:33.851151 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:33.851129 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44b6h\" (UniqueName: \"kubernetes.io/projected/7517a2fd-6dab-41c1-9637-56f0c101f9b7-kube-api-access-44b6h\") pod \"maas-controller-6d64d87758-5ksrd\" (UID: \"7517a2fd-6dab-41c1-9637-56f0c101f9b7\") " pod="opendatahub/maas-controller-6d64d87758-5ksrd" Jul 22 19:05:34.019627 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:34.019585 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d64d87758-5ksrd" Jul 22 19:05:34.146036 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:34.146005 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/905f4e50-cd2a-48cb-81ae-c5d257e11665-webhook-cert\") pod \"maas-controller-66fdbfd797-h4qgm\" (UID: \"905f4e50-cd2a-48cb-81ae-c5d257e11665\") " pod="opendatahub/maas-controller-66fdbfd797-h4qgm" Jul 22 19:05:34.148707 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:34.148683 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/905f4e50-cd2a-48cb-81ae-c5d257e11665-webhook-cert\") pod \"maas-controller-66fdbfd797-h4qgm\" (UID: \"905f4e50-cd2a-48cb-81ae-c5d257e11665\") " pod="opendatahub/maas-controller-66fdbfd797-h4qgm" Jul 22 19:05:34.170341 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:34.170313 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d64d87758-5ksrd"] Jul 22 19:05:34.435091 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:34.435005 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fdbfd797-h4qgm" Jul 22 19:05:34.491798 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:34.491717 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d64d87758-5ksrd" event={"ID":"7517a2fd-6dab-41c1-9637-56f0c101f9b7","Type":"ContainerStarted","Data":"a8786617edf55c008625ac16c6289c95603412d2d86598da90f6dcf1d4e30aa6"} Jul 22 19:05:34.576281 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:34.576256 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66fdbfd797-h4qgm"] Jul 22 19:05:34.577462 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:05:34.577429 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905f4e50_cd2a_48cb_81ae_c5d257e11665.slice/crio-0aa9f85edad441dd25925bdd0334e6869e0db2c881099bf1226f65fea7e66848 WatchSource:0}: Error finding container 0aa9f85edad441dd25925bdd0334e6869e0db2c881099bf1226f65fea7e66848: Status 404 returned error can't find the container with id 0aa9f85edad441dd25925bdd0334e6869e0db2c881099bf1226f65fea7e66848 Jul 22 19:05:35.503072 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:35.502997 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fdbfd797-h4qgm" event={"ID":"905f4e50-cd2a-48cb-81ae-c5d257e11665","Type":"ContainerStarted","Data":"0aa9f85edad441dd25925bdd0334e6869e0db2c881099bf1226f65fea7e66848"} Jul 22 19:05:37.516916 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:37.516881 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fdbfd797-h4qgm" event={"ID":"905f4e50-cd2a-48cb-81ae-c5d257e11665","Type":"ContainerStarted","Data":"b9a3f4340705e85b334528426f8a5fb3e741841c217287a5680ef95b56a132f4"} Jul 22 19:05:37.517381 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:37.516948 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66fdbfd797-h4qgm" Jul 22 19:05:37.518519 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:37.518475 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d64d87758-5ksrd" event={"ID":"7517a2fd-6dab-41c1-9637-56f0c101f9b7","Type":"ContainerStarted","Data":"d898a807dcd2bb5226fe6089b8e93c54ad4d3393113a5435c94f36fd870fe2e3"} Jul 22 19:05:37.518647 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:37.518542 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d64d87758-5ksrd" Jul 22 19:05:37.536703 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:37.536646 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66fdbfd797-h4qgm" podStartSLOduration=2.30137423 podStartE2EDuration="4.536626907s" totalImagesPullingTime="2.235252677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:05:37.535198774 +0000 UTC m=+540.447362843" watchObservedRunningTime="2026-07-22 19:05:37.536626907 +0000 UTC m=+540.448790975" Jul 22 19:05:37.555551 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:37.555483 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d64d87758-5ksrd" podStartSLOduration=1.9161747930000002 podStartE2EDuration="4.555466861s" totalImagesPullingTime="2.639292068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:05:37.553258061 +0000 UTC m=+540.465422113" watchObservedRunningTime="2026-07-22 19:05:37.555466861 +0000 UTC m=+540.467630928" Jul 22 19:05:48.526734 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:48.526704 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66fdbfd797-h4qgm" Jul 22 19:05:48.527155 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:48.526780 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d64d87758-5ksrd" Jul 22 19:05:48.596903 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:48.596869 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66fdbfd797-h4qgm"] Jul 22 19:05:48.597120 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:48.597092 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-66fdbfd797-h4qgm" podUID="905f4e50-cd2a-48cb-81ae-c5d257e11665" containerName="manager" containerID="cri-o://b9a3f4340705e85b334528426f8a5fb3e741841c217287a5680ef95b56a132f4" gracePeriod=30 Jul 22 19:05:48.852750 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:48.852721 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fdbfd797-h4qgm" Jul 22 19:05:48.989548 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:48.989506 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/905f4e50-cd2a-48cb-81ae-c5d257e11665-webhook-cert\") pod \"905f4e50-cd2a-48cb-81ae-c5d257e11665\" (UID: \"905f4e50-cd2a-48cb-81ae-c5d257e11665\") " Jul 22 19:05:48.989749 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:48.989619 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrdtg\" (UniqueName: \"kubernetes.io/projected/905f4e50-cd2a-48cb-81ae-c5d257e11665-kube-api-access-rrdtg\") pod \"905f4e50-cd2a-48cb-81ae-c5d257e11665\" (UID: \"905f4e50-cd2a-48cb-81ae-c5d257e11665\") " Jul 22 19:05:48.991954 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:48.991914 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905f4e50-cd2a-48cb-81ae-c5d257e11665-kube-api-access-rrdtg" (OuterVolumeSpecName: "kube-api-access-rrdtg") pod "905f4e50-cd2a-48cb-81ae-c5d257e11665" (UID: "905f4e50-cd2a-48cb-81ae-c5d257e11665"). InnerVolumeSpecName "kube-api-access-rrdtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:05:48.992102 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:48.991991 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/905f4e50-cd2a-48cb-81ae-c5d257e11665-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "905f4e50-cd2a-48cb-81ae-c5d257e11665" (UID: "905f4e50-cd2a-48cb-81ae-c5d257e11665"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:05:49.091115 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:49.091015 2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/905f4e50-cd2a-48cb-81ae-c5d257e11665-webhook-cert\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:05:49.091115 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:49.091056 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrdtg\" (UniqueName: \"kubernetes.io/projected/905f4e50-cd2a-48cb-81ae-c5d257e11665-kube-api-access-rrdtg\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:05:49.589224 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:49.589180 2583 generic.go:356] "Generic (PLEG): container finished" podID="905f4e50-cd2a-48cb-81ae-c5d257e11665" containerID="b9a3f4340705e85b334528426f8a5fb3e741841c217287a5680ef95b56a132f4" exitCode=0 Jul 22 19:05:49.589677 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:49.589265 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fdbfd797-h4qgm" Jul 22 19:05:49.589677 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:49.589262 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fdbfd797-h4qgm" event={"ID":"905f4e50-cd2a-48cb-81ae-c5d257e11665","Type":"ContainerDied","Data":"b9a3f4340705e85b334528426f8a5fb3e741841c217287a5680ef95b56a132f4"} Jul 22 19:05:49.589677 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:49.589320 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fdbfd797-h4qgm" event={"ID":"905f4e50-cd2a-48cb-81ae-c5d257e11665","Type":"ContainerDied","Data":"0aa9f85edad441dd25925bdd0334e6869e0db2c881099bf1226f65fea7e66848"} Jul 22 19:05:49.589677 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:49.589341 2583 scope.go:117] "RemoveContainer" containerID="b9a3f4340705e85b334528426f8a5fb3e741841c217287a5680ef95b56a132f4" Jul 22 19:05:49.599553 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:49.599374 2583 scope.go:117] "RemoveContainer" containerID="b9a3f4340705e85b334528426f8a5fb3e741841c217287a5680ef95b56a132f4" Jul 22 19:05:49.599771 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:05:49.599741 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a3f4340705e85b334528426f8a5fb3e741841c217287a5680ef95b56a132f4\": container with ID starting with b9a3f4340705e85b334528426f8a5fb3e741841c217287a5680ef95b56a132f4 not found: ID does not exist" containerID="b9a3f4340705e85b334528426f8a5fb3e741841c217287a5680ef95b56a132f4" Jul 22 19:05:49.599833 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:49.599782 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a3f4340705e85b334528426f8a5fb3e741841c217287a5680ef95b56a132f4"} err="failed to get container status \"b9a3f4340705e85b334528426f8a5fb3e741841c217287a5680ef95b56a132f4\": rpc error: code = NotFound desc = could not find container \"b9a3f4340705e85b334528426f8a5fb3e741841c217287a5680ef95b56a132f4\": container with ID starting with b9a3f4340705e85b334528426f8a5fb3e741841c217287a5680ef95b56a132f4 not found: ID does not exist" Jul 22 19:05:49.606844 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:49.606817 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66fdbfd797-h4qgm"] Jul 22 19:05:49.611851 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:49.611822 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66fdbfd797-h4qgm"] Jul 22 19:05:51.550338 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:05:51.550306 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="905f4e50-cd2a-48cb-81ae-c5d257e11665" path="/var/lib/kubelet/pods/905f4e50-cd2a-48cb-81ae-c5d257e11665/volumes" Jul 22 19:06:10.230200 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.230164 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57bfdff96-9487z"] Jul 22 19:06:10.230711 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.230660 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="905f4e50-cd2a-48cb-81ae-c5d257e11665" containerName="manager" Jul 22 19:06:10.230711 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.230675 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="905f4e50-cd2a-48cb-81ae-c5d257e11665" containerName="manager" Jul 22 19:06:10.230785 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.230753 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="905f4e50-cd2a-48cb-81ae-c5d257e11665" containerName="manager" Jul 22 19:06:10.236451 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.236432 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57bfdff96-9487z" Jul 22 19:06:10.238788 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.238769 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 22 19:06:10.247005 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.246979 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57bfdff96-9487z"] Jul 22 19:06:10.291272 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.291235 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24004b0e-bb1c-4af6-aec6-86b7b16e9621-maas-api-tls\") pod \"maas-api-57bfdff96-9487z\" (UID: \"24004b0e-bb1c-4af6-aec6-86b7b16e9621\") " pod="odh-ai-gateway-infra/maas-api-57bfdff96-9487z" Jul 22 19:06:10.291272 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.291277 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9htz5\" (UniqueName: \"kubernetes.io/projected/24004b0e-bb1c-4af6-aec6-86b7b16e9621-kube-api-access-9htz5\") pod \"maas-api-57bfdff96-9487z\" (UID: \"24004b0e-bb1c-4af6-aec6-86b7b16e9621\") " pod="odh-ai-gateway-infra/maas-api-57bfdff96-9487z" Jul 22 19:06:10.291475 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.291357 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24004b0e-bb1c-4af6-aec6-86b7b16e9621-openshift-service-ca\") pod \"maas-api-57bfdff96-9487z\" (UID: \"24004b0e-bb1c-4af6-aec6-86b7b16e9621\") " pod="odh-ai-gateway-infra/maas-api-57bfdff96-9487z" Jul 22 19:06:10.392592 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.392562 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24004b0e-bb1c-4af6-aec6-86b7b16e9621-maas-api-tls\") pod \"maas-api-57bfdff96-9487z\" (UID: \"24004b0e-bb1c-4af6-aec6-86b7b16e9621\") " pod="odh-ai-gateway-infra/maas-api-57bfdff96-9487z" Jul 22 19:06:10.392592 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.392597 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9htz5\" (UniqueName: \"kubernetes.io/projected/24004b0e-bb1c-4af6-aec6-86b7b16e9621-kube-api-access-9htz5\") pod \"maas-api-57bfdff96-9487z\" (UID: \"24004b0e-bb1c-4af6-aec6-86b7b16e9621\") " pod="odh-ai-gateway-infra/maas-api-57bfdff96-9487z" Jul 22 19:06:10.392811 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.392628 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24004b0e-bb1c-4af6-aec6-86b7b16e9621-openshift-service-ca\") pod \"maas-api-57bfdff96-9487z\" (UID: \"24004b0e-bb1c-4af6-aec6-86b7b16e9621\") " pod="odh-ai-gateway-infra/maas-api-57bfdff96-9487z" Jul 22 19:06:10.393254 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.393233 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24004b0e-bb1c-4af6-aec6-86b7b16e9621-openshift-service-ca\") pod \"maas-api-57bfdff96-9487z\" (UID: \"24004b0e-bb1c-4af6-aec6-86b7b16e9621\") " pod="odh-ai-gateway-infra/maas-api-57bfdff96-9487z" Jul 22 19:06:10.395306 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.395272 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24004b0e-bb1c-4af6-aec6-86b7b16e9621-maas-api-tls\") pod \"maas-api-57bfdff96-9487z\" (UID: \"24004b0e-bb1c-4af6-aec6-86b7b16e9621\") " pod="odh-ai-gateway-infra/maas-api-57bfdff96-9487z" Jul 22 19:06:10.404977 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.404945 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9htz5\" (UniqueName: \"kubernetes.io/projected/24004b0e-bb1c-4af6-aec6-86b7b16e9621-kube-api-access-9htz5\") pod \"maas-api-57bfdff96-9487z\" (UID: \"24004b0e-bb1c-4af6-aec6-86b7b16e9621\") " pod="odh-ai-gateway-infra/maas-api-57bfdff96-9487z" Jul 22 19:06:10.548782 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.548748 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57bfdff96-9487z" Jul 22 19:06:10.718969 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.718939 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57bfdff96-9487z"] Jul 22 19:06:10.721590 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:06:10.721555 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24004b0e_bb1c_4af6_aec6_86b7b16e9621.slice/crio-f42102cd87cadddd6a8f2c5510afc86dd007a7d37d27f8d843b143fcdf79ead2 WatchSource:0}: Error finding container f42102cd87cadddd6a8f2c5510afc86dd007a7d37d27f8d843b143fcdf79ead2: Status 404 returned error can't find the container with id f42102cd87cadddd6a8f2c5510afc86dd007a7d37d27f8d843b143fcdf79ead2 Jul 22 19:06:10.872282 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.872205 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-rr9rg"] Jul 22 19:06:10.877574 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.877553 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-rr9rg" Jul 22 19:06:10.880465 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.880435 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hsqpg\"" Jul 22 19:06:10.880635 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.880442 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 22 19:06:10.887541 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.887515 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-rr9rg"] Jul 22 19:06:10.898075 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.898034 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31cb3166-0d51-40f5-98ad-aa881f0b6ffb-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-rr9rg\" (UID: \"31cb3166-0d51-40f5-98ad-aa881f0b6ffb\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-rr9rg" Jul 22 19:06:10.898243 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.898095 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5ctf\" (UniqueName: \"kubernetes.io/projected/31cb3166-0d51-40f5-98ad-aa881f0b6ffb-kube-api-access-r5ctf\") pod \"payload-pre-processing-7677b774bb-rr9rg\" (UID: \"31cb3166-0d51-40f5-98ad-aa881f0b6ffb\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-rr9rg" Jul 22 19:06:10.999316 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.999282 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ctf\" (UniqueName: \"kubernetes.io/projected/31cb3166-0d51-40f5-98ad-aa881f0b6ffb-kube-api-access-r5ctf\") pod \"payload-pre-processing-7677b774bb-rr9rg\" (UID: \"31cb3166-0d51-40f5-98ad-aa881f0b6ffb\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-rr9rg" Jul 22 19:06:10.999483 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:10.999405 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31cb3166-0d51-40f5-98ad-aa881f0b6ffb-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-rr9rg\" (UID: \"31cb3166-0d51-40f5-98ad-aa881f0b6ffb\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-rr9rg" Jul 22 19:06:11.000138 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:11.000116 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31cb3166-0d51-40f5-98ad-aa881f0b6ffb-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-rr9rg\" (UID: \"31cb3166-0d51-40f5-98ad-aa881f0b6ffb\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-rr9rg" Jul 22 19:06:11.008064 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:11.008034 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5ctf\" (UniqueName: \"kubernetes.io/projected/31cb3166-0d51-40f5-98ad-aa881f0b6ffb-kube-api-access-r5ctf\") pod \"payload-pre-processing-7677b774bb-rr9rg\" (UID: \"31cb3166-0d51-40f5-98ad-aa881f0b6ffb\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-rr9rg" Jul 22 19:06:11.191200 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:11.191103 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-rr9rg" Jul 22 19:06:11.331125 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:11.331096 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-rr9rg"] Jul 22 19:06:11.333064 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:06:11.333040 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31cb3166_0d51_40f5_98ad_aa881f0b6ffb.slice/crio-0777a17bffcd3cbeaa911b9f505848a6f6ac06b680b782bdd0204106218f19a0 WatchSource:0}: Error finding container 0777a17bffcd3cbeaa911b9f505848a6f6ac06b680b782bdd0204106218f19a0: Status 404 returned error can't find the container with id 0777a17bffcd3cbeaa911b9f505848a6f6ac06b680b782bdd0204106218f19a0 Jul 22 19:06:11.715319 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:11.715283 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-rr9rg" event={"ID":"31cb3166-0d51-40f5-98ad-aa881f0b6ffb","Type":"ContainerStarted","Data":"0777a17bffcd3cbeaa911b9f505848a6f6ac06b680b782bdd0204106218f19a0"} Jul 22 19:06:11.716433 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:11.716409 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57bfdff96-9487z" event={"ID":"24004b0e-bb1c-4af6-aec6-86b7b16e9621","Type":"ContainerStarted","Data":"f42102cd87cadddd6a8f2c5510afc86dd007a7d37d27f8d843b143fcdf79ead2"} Jul 22 19:06:13.729774 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:13.729723 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-rr9rg" event={"ID":"31cb3166-0d51-40f5-98ad-aa881f0b6ffb","Type":"ContainerStarted","Data":"135f8a022fa7d11cb19337516f04f002f1006702d2e7eeda3f46c40aab0aceb5"} Jul 22 19:06:13.730221 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:13.729799 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-rr9rg" Jul 22 19:06:13.731270 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:13.731242 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57bfdff96-9487z" event={"ID":"24004b0e-bb1c-4af6-aec6-86b7b16e9621","Type":"ContainerStarted","Data":"ac4c49aa560e44d3e46905c36d054ff7339df05a1bf19240d93892b7269a521c"} Jul 22 19:06:13.731389 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:13.731374 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57bfdff96-9487z" Jul 22 19:06:13.746588 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:13.746538 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-rr9rg" podStartSLOduration=2.025888352 podStartE2EDuration="3.746523086s" totalImagesPullingTime="1.720634734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:06:13.744730499 +0000 UTC m=+576.656894815" watchObservedRunningTime="2026-07-22 19:06:13.746523086 +0000 UTC m=+576.658687145" Jul 22 19:06:13.761037 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:13.760983 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57bfdff96-9487z" podStartSLOduration=1.433676449 podStartE2EDuration="3.760964449s" totalImagesPullingTime="2.327288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:06:13.76044919 +0000 UTC m=+576.672613271" watchObservedRunningTime="2026-07-22 19:06:13.760964449 +0000 UTC m=+576.673128518" Jul 22 19:06:19.742137 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:19.742106 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57bfdff96-9487z" Jul 22 19:06:24.738972 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:24.738939 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-rr9rg" Jul 22 19:06:29.461595 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.461551 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw"] Jul 22 19:06:29.465939 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.465916 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.469637 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.469611 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 22 19:06:29.469637 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.469612 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 19:06:29.469804 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.469618 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qbfdn\"" Jul 22 19:06:29.469804 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.469700 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 19:06:29.475258 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.475238 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw"] Jul 22 19:06:29.573993 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.573952 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9805fea0-87e3-40fd-9534-feea14fb01a1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.574168 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.574023 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mww2g\" (UniqueName: \"kubernetes.io/projected/9805fea0-87e3-40fd-9534-feea14fb01a1-kube-api-access-mww2g\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.574168 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.574096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9805fea0-87e3-40fd-9534-feea14fb01a1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.574168 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.574146 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9805fea0-87e3-40fd-9534-feea14fb01a1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.574168 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.574164 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9805fea0-87e3-40fd-9534-feea14fb01a1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.574309 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.574232 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9805fea0-87e3-40fd-9534-feea14fb01a1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.674892 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.674850 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9805fea0-87e3-40fd-9534-feea14fb01a1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.674892 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.674894 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9805fea0-87e3-40fd-9534-feea14fb01a1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.675167 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.674922 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9805fea0-87e3-40fd-9534-feea14fb01a1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.675167 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.674951 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9805fea0-87e3-40fd-9534-feea14fb01a1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.675167 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.674994 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mww2g\" (UniqueName: \"kubernetes.io/projected/9805fea0-87e3-40fd-9534-feea14fb01a1-kube-api-access-mww2g\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.675167 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.675029 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9805fea0-87e3-40fd-9534-feea14fb01a1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.675381 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.675337 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9805fea0-87e3-40fd-9534-feea14fb01a1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.675381 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.675350 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9805fea0-87e3-40fd-9534-feea14fb01a1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.675474 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.675432 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9805fea0-87e3-40fd-9534-feea14fb01a1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.677483 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.677457 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9805fea0-87e3-40fd-9534-feea14fb01a1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.677713 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.677695 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9805fea0-87e3-40fd-9534-feea14fb01a1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.682882 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.682861 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mww2g\" (UniqueName: \"kubernetes.io/projected/9805fea0-87e3-40fd-9534-feea14fb01a1-kube-api-access-mww2g\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw\" (UID: \"9805fea0-87e3-40fd-9534-feea14fb01a1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.777590 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.777554 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:29.928865 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:29.928840 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw"] Jul 22 19:06:29.930576 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:06:29.930545 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9805fea0_87e3_40fd_9534_feea14fb01a1.slice/crio-1d08aa273c55a5700a289e2b41230a7bd82378764757023cddf452e029ab6503 WatchSource:0}: Error finding container 1d08aa273c55a5700a289e2b41230a7bd82378764757023cddf452e029ab6503: Status 404 returned error can't find the container with id 1d08aa273c55a5700a289e2b41230a7bd82378764757023cddf452e029ab6503 Jul 22 19:06:30.839440 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:30.839403 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" event={"ID":"9805fea0-87e3-40fd-9534-feea14fb01a1","Type":"ContainerStarted","Data":"1d08aa273c55a5700a289e2b41230a7bd82378764757023cddf452e029ab6503"} Jul 22 19:06:31.846509 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:31.846461 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" event={"ID":"9805fea0-87e3-40fd-9534-feea14fb01a1","Type":"ContainerStarted","Data":"6c83105b0dd19c9881d4fe9b8362829a0c4f17ab38ff279d4fe46cda5d28987f"} Jul 22 19:06:31.867039 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:31.866987 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" podStartSLOduration=1.817348282 podStartE2EDuration="2.866971401s" totalImagesPullingTime="1.049623119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:06:31.866187687 +0000 UTC m=+594.778351769" watchObservedRunningTime="2026-07-22 19:06:31.866971401 +0000 UTC m=+594.779135468" Jul 22 19:06:37.509021 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:37.508996 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:06:37.509579 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:37.509542 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:06:39.778415 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:39.778373 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:39.780757 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:39.778440 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:39.795510 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:39.795473 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:39.903321 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:39.903291 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw" Jul 22 19:06:44.647257 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.645736 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m"] Jul 22 19:06:44.650123 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.650097 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.655056 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.655031 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 22 19:06:44.657672 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.657647 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m"] Jul 22 19:06:44.717974 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.717938 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.717974 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.717976 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.718183 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.718054 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.718183 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.718165 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.718266 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.718188 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvqxm\" (UniqueName: \"kubernetes.io/projected/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-kube-api-access-pvqxm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.718266 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.718211 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.819445 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.819405 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.819445 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.819448 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvqxm\" (UniqueName: \"kubernetes.io/projected/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-kube-api-access-pvqxm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.819692 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.819483 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.819692 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.819568 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.819692 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.819599 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.819692 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.819649 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.819988 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.819966 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.820062 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.820013 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.820062 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.820035 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.821866 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.821840 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.822245 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.822223 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.827447 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.827421 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvqxm\" (UniqueName: \"kubernetes.io/projected/b59b394f-42c8-4a7e-a3e1-4cb9d378721d-kube-api-access-pvqxm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4s49m\" (UID: \"b59b394f-42c8-4a7e-a3e1-4cb9d378721d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:44.968362 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:44.968268 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:45.112778 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:45.112751 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m"] Jul 22 19:06:45.117556 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:06:45.117521 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59b394f_42c8_4a7e_a3e1_4cb9d378721d.slice/crio-59534c92aa83cf1c4f145ebcbee6acb67d6cb7793f5f2b4780cf249ba4e78dc8 WatchSource:0}: Error finding container 59534c92aa83cf1c4f145ebcbee6acb67d6cb7793f5f2b4780cf249ba4e78dc8: Status 404 returned error can't find the container with id 59534c92aa83cf1c4f145ebcbee6acb67d6cb7793f5f2b4780cf249ba4e78dc8 Jul 22 19:06:45.933400 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:45.933359 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" event={"ID":"b59b394f-42c8-4a7e-a3e1-4cb9d378721d","Type":"ContainerStarted","Data":"d2d09f0aa14a3b6888f4c520e660adc2193d84c661821cffe29cc8c3bd8b9b5c"} Jul 22 19:06:45.933811 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:45.933417 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" event={"ID":"b59b394f-42c8-4a7e-a3e1-4cb9d378721d","Type":"ContainerStarted","Data":"59534c92aa83cf1c4f145ebcbee6acb67d6cb7793f5f2b4780cf249ba4e78dc8"} Jul 22 19:06:45.950957 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:45.950900 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" podStartSLOduration=1.763142397 podStartE2EDuration="1.950884149s" totalImagesPullingTime="187.741752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:06:45.949653649 +0000 UTC m=+608.861817740" watchObservedRunningTime="2026-07-22 19:06:45.950884149 +0000 UTC m=+608.863048216" Jul 22 19:06:54.968968 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:54.968922 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:54.969580 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:54.969002 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:54.981973 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:54.981941 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:06:54.998889 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:06:54.998859 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4s49m" Jul 22 19:09:05.348584 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.348546 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d64d87758-5ksrd"] Jul 22 19:09:05.349150 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.348777 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6d64d87758-5ksrd" podUID="7517a2fd-6dab-41c1-9637-56f0c101f9b7" containerName="manager" containerID="cri-o://d898a807dcd2bb5226fe6089b8e93c54ad4d3393113a5435c94f36fd870fe2e3" gracePeriod=30 Jul 22 19:09:05.604824 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.604757 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d64d87758-5ksrd" Jul 22 19:09:05.700654 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.700611 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44b6h\" (UniqueName: \"kubernetes.io/projected/7517a2fd-6dab-41c1-9637-56f0c101f9b7-kube-api-access-44b6h\") pod \"7517a2fd-6dab-41c1-9637-56f0c101f9b7\" (UID: \"7517a2fd-6dab-41c1-9637-56f0c101f9b7\") " Jul 22 19:09:05.700852 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.700701 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7517a2fd-6dab-41c1-9637-56f0c101f9b7-webhook-cert\") pod \"7517a2fd-6dab-41c1-9637-56f0c101f9b7\" (UID: \"7517a2fd-6dab-41c1-9637-56f0c101f9b7\") " Jul 22 19:09:05.702974 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.702934 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7517a2fd-6dab-41c1-9637-56f0c101f9b7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7517a2fd-6dab-41c1-9637-56f0c101f9b7" (UID: "7517a2fd-6dab-41c1-9637-56f0c101f9b7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:09:05.703098 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.702974 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7517a2fd-6dab-41c1-9637-56f0c101f9b7-kube-api-access-44b6h" (OuterVolumeSpecName: "kube-api-access-44b6h") pod "7517a2fd-6dab-41c1-9637-56f0c101f9b7" (UID: "7517a2fd-6dab-41c1-9637-56f0c101f9b7"). InnerVolumeSpecName "kube-api-access-44b6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:09:05.750606 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.750572 2583 generic.go:356] "Generic (PLEG): container finished" podID="7517a2fd-6dab-41c1-9637-56f0c101f9b7" containerID="d898a807dcd2bb5226fe6089b8e93c54ad4d3393113a5435c94f36fd870fe2e3" exitCode=0 Jul 22 19:09:05.750606 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.750609 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d64d87758-5ksrd" event={"ID":"7517a2fd-6dab-41c1-9637-56f0c101f9b7","Type":"ContainerDied","Data":"d898a807dcd2bb5226fe6089b8e93c54ad4d3393113a5435c94f36fd870fe2e3"} Jul 22 19:09:05.750799 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.750623 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d64d87758-5ksrd" Jul 22 19:09:05.750799 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.750649 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d64d87758-5ksrd" event={"ID":"7517a2fd-6dab-41c1-9637-56f0c101f9b7","Type":"ContainerDied","Data":"a8786617edf55c008625ac16c6289c95603412d2d86598da90f6dcf1d4e30aa6"} Jul 22 19:09:05.750799 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.750671 2583 scope.go:117] "RemoveContainer" containerID="d898a807dcd2bb5226fe6089b8e93c54ad4d3393113a5435c94f36fd870fe2e3" Jul 22 19:09:05.764190 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.764171 2583 scope.go:117] "RemoveContainer" containerID="d898a807dcd2bb5226fe6089b8e93c54ad4d3393113a5435c94f36fd870fe2e3" Jul 22 19:09:05.764463 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:09:05.764445 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d898a807dcd2bb5226fe6089b8e93c54ad4d3393113a5435c94f36fd870fe2e3\": container with ID starting with d898a807dcd2bb5226fe6089b8e93c54ad4d3393113a5435c94f36fd870fe2e3 not found: ID does not exist" containerID="d898a807dcd2bb5226fe6089b8e93c54ad4d3393113a5435c94f36fd870fe2e3" Jul 22 19:09:05.764541 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.764471 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d898a807dcd2bb5226fe6089b8e93c54ad4d3393113a5435c94f36fd870fe2e3"} err="failed to get container status \"d898a807dcd2bb5226fe6089b8e93c54ad4d3393113a5435c94f36fd870fe2e3\": rpc error: code = NotFound desc = could not find container \"d898a807dcd2bb5226fe6089b8e93c54ad4d3393113a5435c94f36fd870fe2e3\": container with ID starting with d898a807dcd2bb5226fe6089b8e93c54ad4d3393113a5435c94f36fd870fe2e3 not found: ID does not exist" Jul 22 19:09:05.772913 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.772887 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d64d87758-5ksrd"] Jul 22 19:09:05.779085 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.779063 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d64d87758-5ksrd"] Jul 22 19:09:05.801864 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.801839 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44b6h\" (UniqueName: \"kubernetes.io/projected/7517a2fd-6dab-41c1-9637-56f0c101f9b7-kube-api-access-44b6h\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:09:05.801864 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:05.801864 2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7517a2fd-6dab-41c1-9637-56f0c101f9b7-webhook-cert\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:09:07.551992 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:09:07.551959 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7517a2fd-6dab-41c1-9637-56f0c101f9b7" path="/var/lib/kubelet/pods/7517a2fd-6dab-41c1-9637-56f0c101f9b7/volumes" Jul 22 19:10:06.605350 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:06.605268 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d64d87758-626h8"] Jul 22 19:10:06.605769 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:06.605710 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7517a2fd-6dab-41c1-9637-56f0c101f9b7" containerName="manager" Jul 22 19:10:06.605769 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:06.605722 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7517a2fd-6dab-41c1-9637-56f0c101f9b7" containerName="manager" Jul 22 19:10:06.605845 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:06.605797 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7517a2fd-6dab-41c1-9637-56f0c101f9b7" containerName="manager" Jul 22 19:10:06.608014 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:06.607997 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d64d87758-626h8" Jul 22 19:10:06.610848 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:06.610824 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-g2pff\"" Jul 22 19:10:06.611218 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:06.611200 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 19:10:06.611825 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:06.611811 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 19:10:06.628861 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:06.628833 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d64d87758-626h8"] Jul 22 19:10:06.693932 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:06.693895 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrg4t\" (UniqueName: \"kubernetes.io/projected/2d6bf3d0-28ee-4815-af76-f5e64913ee1c-kube-api-access-lrg4t\") pod \"maas-controller-6d64d87758-626h8\" (UID: \"2d6bf3d0-28ee-4815-af76-f5e64913ee1c\") " pod="opendatahub/maas-controller-6d64d87758-626h8" Jul 22 19:10:06.694137 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:06.693959 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d6bf3d0-28ee-4815-af76-f5e64913ee1c-webhook-cert\") pod \"maas-controller-6d64d87758-626h8\" (UID: \"2d6bf3d0-28ee-4815-af76-f5e64913ee1c\") " pod="opendatahub/maas-controller-6d64d87758-626h8" Jul 22 19:10:06.795652 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:06.795613 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrg4t\" (UniqueName: \"kubernetes.io/projected/2d6bf3d0-28ee-4815-af76-f5e64913ee1c-kube-api-access-lrg4t\") pod \"maas-controller-6d64d87758-626h8\" (UID: \"2d6bf3d0-28ee-4815-af76-f5e64913ee1c\") " pod="opendatahub/maas-controller-6d64d87758-626h8" Jul 22 19:10:06.795855 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:06.795674 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d6bf3d0-28ee-4815-af76-f5e64913ee1c-webhook-cert\") pod \"maas-controller-6d64d87758-626h8\" (UID: \"2d6bf3d0-28ee-4815-af76-f5e64913ee1c\") " pod="opendatahub/maas-controller-6d64d87758-626h8" Jul 22 19:10:06.798162 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:06.798139 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d6bf3d0-28ee-4815-af76-f5e64913ee1c-webhook-cert\") pod \"maas-controller-6d64d87758-626h8\" (UID: \"2d6bf3d0-28ee-4815-af76-f5e64913ee1c\") " pod="opendatahub/maas-controller-6d64d87758-626h8" Jul 22 19:10:06.803967 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:06.803938 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrg4t\" (UniqueName: \"kubernetes.io/projected/2d6bf3d0-28ee-4815-af76-f5e64913ee1c-kube-api-access-lrg4t\") pod \"maas-controller-6d64d87758-626h8\" (UID: \"2d6bf3d0-28ee-4815-af76-f5e64913ee1c\") " pod="opendatahub/maas-controller-6d64d87758-626h8" Jul 22 19:10:06.918704 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:06.918606 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d64d87758-626h8" Jul 22 19:10:07.060880 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:07.060847 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d64d87758-626h8"] Jul 22 19:10:07.062883 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:10:07.062856 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6bf3d0_28ee_4815_af76_f5e64913ee1c.slice/crio-2453739279ee9f67d5e910fdc1795c9d6d757449ef76fe2fc852c17cfa8c96cd WatchSource:0}: Error finding container 2453739279ee9f67d5e910fdc1795c9d6d757449ef76fe2fc852c17cfa8c96cd: Status 404 returned error can't find the container with id 2453739279ee9f67d5e910fdc1795c9d6d757449ef76fe2fc852c17cfa8c96cd Jul 22 19:10:07.064286 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:07.064267 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 19:10:07.111741 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:07.111702 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d64d87758-626h8" event={"ID":"2d6bf3d0-28ee-4815-af76-f5e64913ee1c","Type":"ContainerStarted","Data":"2453739279ee9f67d5e910fdc1795c9d6d757449ef76fe2fc852c17cfa8c96cd"} Jul 22 19:10:08.119622 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:08.119586 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d64d87758-626h8" event={"ID":"2d6bf3d0-28ee-4815-af76-f5e64913ee1c","Type":"ContainerStarted","Data":"954ec66e37c02db44d68eff028a652ebf7ae064bee226e0df32f323ac4591b8c"} Jul 22 19:10:08.120072 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:08.119690 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d64d87758-626h8" Jul 22 19:10:08.137616 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:08.137562 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d64d87758-626h8" podStartSLOduration=1.729505238 podStartE2EDuration="2.137545524s" totalImagesPullingTime="408.040286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:10:08.134868888 +0000 UTC m=+811.047032956" watchObservedRunningTime="2026-07-22 19:10:08.137545524 +0000 UTC m=+811.049709606" Jul 22 19:10:19.127245 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:19.127215 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d64d87758-626h8" Jul 22 19:10:23.750804 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:23.750769 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d64d87758-626h8"] Jul 22 19:10:23.751324 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:23.750967 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6d64d87758-626h8" podUID="2d6bf3d0-28ee-4815-af76-f5e64913ee1c" containerName="manager" containerID="cri-o://954ec66e37c02db44d68eff028a652ebf7ae064bee226e0df32f323ac4591b8c" gracePeriod=30 Jul 22 19:10:24.008349 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:24.008276 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d64d87758-626h8" Jul 22 19:10:24.170098 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:24.170064 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrg4t\" (UniqueName: \"kubernetes.io/projected/2d6bf3d0-28ee-4815-af76-f5e64913ee1c-kube-api-access-lrg4t\") pod \"2d6bf3d0-28ee-4815-af76-f5e64913ee1c\" (UID: \"2d6bf3d0-28ee-4815-af76-f5e64913ee1c\") " Jul 22 19:10:24.170282 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:24.170224 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d6bf3d0-28ee-4815-af76-f5e64913ee1c-webhook-cert\") pod \"2d6bf3d0-28ee-4815-af76-f5e64913ee1c\" (UID: \"2d6bf3d0-28ee-4815-af76-f5e64913ee1c\") " Jul 22 19:10:24.172260 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:24.172226 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d6bf3d0-28ee-4815-af76-f5e64913ee1c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2d6bf3d0-28ee-4815-af76-f5e64913ee1c" (UID: "2d6bf3d0-28ee-4815-af76-f5e64913ee1c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:10:24.172384 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:24.172361 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6bf3d0-28ee-4815-af76-f5e64913ee1c-kube-api-access-lrg4t" (OuterVolumeSpecName: "kube-api-access-lrg4t") pod "2d6bf3d0-28ee-4815-af76-f5e64913ee1c" (UID: "2d6bf3d0-28ee-4815-af76-f5e64913ee1c"). InnerVolumeSpecName "kube-api-access-lrg4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:10:24.211260 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:24.211223 2583 generic.go:356] "Generic (PLEG): container finished" podID="2d6bf3d0-28ee-4815-af76-f5e64913ee1c" containerID="954ec66e37c02db44d68eff028a652ebf7ae064bee226e0df32f323ac4591b8c" exitCode=0 Jul 22 19:10:24.211415 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:24.211273 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d64d87758-626h8" Jul 22 19:10:24.211415 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:24.211312 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d64d87758-626h8" event={"ID":"2d6bf3d0-28ee-4815-af76-f5e64913ee1c","Type":"ContainerDied","Data":"954ec66e37c02db44d68eff028a652ebf7ae064bee226e0df32f323ac4591b8c"} Jul 22 19:10:24.211415 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:24.211392 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d64d87758-626h8" event={"ID":"2d6bf3d0-28ee-4815-af76-f5e64913ee1c","Type":"ContainerDied","Data":"2453739279ee9f67d5e910fdc1795c9d6d757449ef76fe2fc852c17cfa8c96cd"} Jul 22 19:10:24.211415 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:24.211414 2583 scope.go:117] "RemoveContainer" containerID="954ec66e37c02db44d68eff028a652ebf7ae064bee226e0df32f323ac4591b8c" Jul 22 19:10:24.221754 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:24.221734 2583 scope.go:117] "RemoveContainer" containerID="954ec66e37c02db44d68eff028a652ebf7ae064bee226e0df32f323ac4591b8c" Jul 22 19:10:24.222032 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:10:24.222012 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954ec66e37c02db44d68eff028a652ebf7ae064bee226e0df32f323ac4591b8c\": container with ID starting with 954ec66e37c02db44d68eff028a652ebf7ae064bee226e0df32f323ac4591b8c not found: ID does not exist" containerID="954ec66e37c02db44d68eff028a652ebf7ae064bee226e0df32f323ac4591b8c" Jul 22 19:10:24.222094 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:24.222041 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954ec66e37c02db44d68eff028a652ebf7ae064bee226e0df32f323ac4591b8c"} err="failed to get container status \"954ec66e37c02db44d68eff028a652ebf7ae064bee226e0df32f323ac4591b8c\": rpc error: code = NotFound desc = could not find container \"954ec66e37c02db44d68eff028a652ebf7ae064bee226e0df32f323ac4591b8c\": container with ID starting with 954ec66e37c02db44d68eff028a652ebf7ae064bee226e0df32f323ac4591b8c not found: ID does not exist" Jul 22 19:10:24.232687 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:24.232655 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d64d87758-626h8"] Jul 22 19:10:24.236253 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:24.236229 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d64d87758-626h8"] Jul 22 19:10:24.271105 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:24.271080 2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d6bf3d0-28ee-4815-af76-f5e64913ee1c-webhook-cert\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:10:24.271105 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:24.271104 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrg4t\" (UniqueName: \"kubernetes.io/projected/2d6bf3d0-28ee-4815-af76-f5e64913ee1c-kube-api-access-lrg4t\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:10:25.018114 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.018081 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d64d87758-2fzpf"] Jul 22 19:10:25.018552 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.018515 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d6bf3d0-28ee-4815-af76-f5e64913ee1c" containerName="manager" Jul 22 19:10:25.018552 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.018527 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d6bf3d0-28ee-4815-af76-f5e64913ee1c" containerName="manager" Jul 22 19:10:25.018645 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.018606 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d6bf3d0-28ee-4815-af76-f5e64913ee1c" containerName="manager" Jul 22 19:10:25.021769 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.021754 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d64d87758-2fzpf" Jul 22 19:10:25.026996 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.026972 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-g2pff\"" Jul 22 19:10:25.027122 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.026995 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 19:10:25.027122 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.026999 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 19:10:25.034263 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.034239 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d64d87758-2fzpf"] Jul 22 19:10:25.179370 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.179333 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzkrs\" (UniqueName: \"kubernetes.io/projected/d59b3764-c264-4cf1-8d6c-dadc1c47229f-kube-api-access-vzkrs\") pod \"maas-controller-6d64d87758-2fzpf\" (UID: \"d59b3764-c264-4cf1-8d6c-dadc1c47229f\") " pod="opendatahub/maas-controller-6d64d87758-2fzpf" Jul 22 19:10:25.179574 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.179414 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d59b3764-c264-4cf1-8d6c-dadc1c47229f-webhook-cert\") pod \"maas-controller-6d64d87758-2fzpf\" (UID: \"d59b3764-c264-4cf1-8d6c-dadc1c47229f\") " pod="opendatahub/maas-controller-6d64d87758-2fzpf" Jul 22 19:10:25.280223 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.280126 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d59b3764-c264-4cf1-8d6c-dadc1c47229f-webhook-cert\") pod \"maas-controller-6d64d87758-2fzpf\" (UID: \"d59b3764-c264-4cf1-8d6c-dadc1c47229f\") " pod="opendatahub/maas-controller-6d64d87758-2fzpf" Jul 22 19:10:25.280223 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.280216 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzkrs\" (UniqueName: \"kubernetes.io/projected/d59b3764-c264-4cf1-8d6c-dadc1c47229f-kube-api-access-vzkrs\") pod \"maas-controller-6d64d87758-2fzpf\" (UID: \"d59b3764-c264-4cf1-8d6c-dadc1c47229f\") " pod="opendatahub/maas-controller-6d64d87758-2fzpf" Jul 22 19:10:25.282800 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.282774 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d59b3764-c264-4cf1-8d6c-dadc1c47229f-webhook-cert\") pod \"maas-controller-6d64d87758-2fzpf\" (UID: \"d59b3764-c264-4cf1-8d6c-dadc1c47229f\") " pod="opendatahub/maas-controller-6d64d87758-2fzpf" Jul 22 19:10:25.289205 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.289181 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzkrs\" (UniqueName: \"kubernetes.io/projected/d59b3764-c264-4cf1-8d6c-dadc1c47229f-kube-api-access-vzkrs\") pod \"maas-controller-6d64d87758-2fzpf\" (UID: \"d59b3764-c264-4cf1-8d6c-dadc1c47229f\") " pod="opendatahub/maas-controller-6d64d87758-2fzpf" Jul 22 19:10:25.333447 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.333413 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d64d87758-2fzpf" Jul 22 19:10:25.476650 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.476623 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d64d87758-2fzpf"] Jul 22 19:10:25.477573 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:10:25.477551 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59b3764_c264_4cf1_8d6c_dadc1c47229f.slice/crio-9a9220c54c8d68c7eacdaf80bf60823ba38f0374115236116f0fa030a08bdb68 WatchSource:0}: Error finding container 9a9220c54c8d68c7eacdaf80bf60823ba38f0374115236116f0fa030a08bdb68: Status 404 returned error can't find the container with id 9a9220c54c8d68c7eacdaf80bf60823ba38f0374115236116f0fa030a08bdb68 Jul 22 19:10:25.551369 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:25.551288 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d6bf3d0-28ee-4815-af76-f5e64913ee1c" path="/var/lib/kubelet/pods/2d6bf3d0-28ee-4815-af76-f5e64913ee1c/volumes" Jul 22 19:10:26.226030 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:26.225991 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d64d87758-2fzpf" event={"ID":"d59b3764-c264-4cf1-8d6c-dadc1c47229f","Type":"ContainerStarted","Data":"c137636a1c8d927752d1c29a991c32a3ae6d8fb6bdd2946c8eb378428454d086"} Jul 22 19:10:26.226421 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:26.226149 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d64d87758-2fzpf" Jul 22 19:10:26.226421 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:26.226191 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d64d87758-2fzpf" event={"ID":"d59b3764-c264-4cf1-8d6c-dadc1c47229f","Type":"ContainerStarted","Data":"9a9220c54c8d68c7eacdaf80bf60823ba38f0374115236116f0fa030a08bdb68"} Jul 22 19:10:26.267635 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:26.267593 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d64d87758-2fzpf" podStartSLOduration=1.7424387239999999 podStartE2EDuration="2.267575896s" totalImagesPullingTime="525.137172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:10:26.265733617 +0000 UTC m=+829.177897685" watchObservedRunningTime="2026-07-22 19:10:26.267575896 +0000 UTC m=+829.179739964" Jul 22 19:10:37.234699 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:10:37.234670 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d64d87758-2fzpf" Jul 22 19:11:37.549558 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:11:37.549518 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:11:37.553117 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:11:37.553095 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:11:37.596357 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:11:37.596323 2583 scope.go:117] "RemoveContainer" containerID="4909b105fb72b6e55e33848eefd172f86ea7493e563283d44a84da1b751aa90d" Jul 22 19:15:00.152626 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:15:00.152588 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745795-d5x4x"] Jul 22 19:15:00.156326 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:15:00.156304 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745795-d5x4x" Jul 22 19:15:00.159071 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:15:00.159050 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jhv4j\"" Jul 22 19:15:00.166167 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:15:00.166144 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745795-d5x4x" Jul 22 19:15:00.174774 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:15:00.174730 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745795-d5x4x"] Jul 22 19:15:00.324087 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:15:00.324039 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745795-d5x4x"] Jul 22 19:15:00.330988 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:15:00.330959 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b20b3c_fb49_4543_a19e_edbf415574d3.slice/crio-a7c2a44375e416dd49f8f148ed1debad8406a0c6e356ff722f45b4e261bf17c2 WatchSource:0}: Error finding container a7c2a44375e416dd49f8f148ed1debad8406a0c6e356ff722f45b4e261bf17c2: Status 404 returned error can't find the container with id a7c2a44375e416dd49f8f148ed1debad8406a0c6e356ff722f45b4e261bf17c2 Jul 22 19:15:00.843765 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:15:00.843722 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745795-d5x4x" event={"ID":"80b20b3c-fb49-4543-a19e-edbf415574d3","Type":"ContainerStarted","Data":"a7c2a44375e416dd49f8f148ed1debad8406a0c6e356ff722f45b4e261bf17c2"} Jul 22 19:15:05.875618 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:15:05.875573 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745795-d5x4x" event={"ID":"80b20b3c-fb49-4543-a19e-edbf415574d3","Type":"ContainerStarted","Data":"365c40792e29d5342987130626b83b9a5ec03c70f44067fb201fe0052086418b"} Jul 22 19:15:05.893147 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:15:05.893095 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745795-d5x4x" podStartSLOduration=1.261585071 podStartE2EDuration="5.89307805s" totalImagesPullingTime="4.631492979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:15:05.890370502 +0000 UTC m=+1108.802534570" watchObservedRunningTime="2026-07-22 19:15:05.89307805 +0000 UTC m=+1108.805242129" Jul 22 19:15:06.883274 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:15:06.883213 2583 generic.go:356] "Generic (PLEG): container finished" podID="80b20b3c-fb49-4543-a19e-edbf415574d3" containerID="365c40792e29d5342987130626b83b9a5ec03c70f44067fb201fe0052086418b" exitCode=0 Jul 22 19:15:06.883752 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:15:06.883317 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745795-d5x4x" event={"ID":"80b20b3c-fb49-4543-a19e-edbf415574d3","Type":"ContainerDied","Data":"365c40792e29d5342987130626b83b9a5ec03c70f44067fb201fe0052086418b"} Jul 22 19:15:08.036019 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:15:08.035995 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745795-d5x4x" Jul 22 19:15:08.897929 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:15:08.897889 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745795-d5x4x" event={"ID":"80b20b3c-fb49-4543-a19e-edbf415574d3","Type":"ContainerDied","Data":"a7c2a44375e416dd49f8f148ed1debad8406a0c6e356ff722f45b4e261bf17c2"} Jul 22 19:15:08.898123 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:15:08.897939 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7c2a44375e416dd49f8f148ed1debad8406a0c6e356ff722f45b4e261bf17c2" Jul 22 19:15:08.898123 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:15:08.897906 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745795-d5x4x" Jul 22 19:16:37.594420 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:16:37.594382 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:16:37.599961 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:16:37.599940 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:21:37.633979 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:21:37.633951 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:21:37.642352 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:21:37.642331 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:23:39.061172 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.061096 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq"] Jul 22 19:23:39.061697 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.061571 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b20b3c-fb49-4543-a19e-edbf415574d3" containerName="cleanup" Jul 22 19:23:39.061697 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.061585 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b20b3c-fb49-4543-a19e-edbf415574d3" containerName="cleanup" Jul 22 19:23:39.061697 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.061655 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b20b3c-fb49-4543-a19e-edbf415574d3" containerName="cleanup" Jul 22 19:23:39.064751 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.064730 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" Jul 22 19:23:39.067379 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.067358 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t4d8t\"" Jul 22 19:23:39.067520 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.067381 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-cd4f9eba-serving-cert\"" Jul 22 19:23:39.074572 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.074304 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq"] Jul 22 19:23:39.095129 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.095096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d410e9-49c3-41fb-a7eb-3fd63914be70-openshift-service-ca\") pod \"maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq\" (UID: \"06d410e9-49c3-41fb-a7eb-3fd63914be70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" Jul 22 19:23:39.095291 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.095134 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nkws\" (UniqueName: \"kubernetes.io/projected/06d410e9-49c3-41fb-a7eb-3fd63914be70-kube-api-access-5nkws\") pod \"maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq\" (UID: \"06d410e9-49c3-41fb-a7eb-3fd63914be70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" Jul 22 19:23:39.095291 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.095240 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d410e9-49c3-41fb-a7eb-3fd63914be70-maas-api-tls\") pod \"maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq\" (UID: \"06d410e9-49c3-41fb-a7eb-3fd63914be70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" Jul 22 19:23:39.196408 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.196368 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d410e9-49c3-41fb-a7eb-3fd63914be70-maas-api-tls\") pod \"maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq\" (UID: \"06d410e9-49c3-41fb-a7eb-3fd63914be70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" Jul 22 19:23:39.196602 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.196432 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d410e9-49c3-41fb-a7eb-3fd63914be70-openshift-service-ca\") pod \"maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq\" (UID: \"06d410e9-49c3-41fb-a7eb-3fd63914be70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" Jul 22 19:23:39.196602 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.196469 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nkws\" (UniqueName: \"kubernetes.io/projected/06d410e9-49c3-41fb-a7eb-3fd63914be70-kube-api-access-5nkws\") pod \"maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq\" (UID: \"06d410e9-49c3-41fb-a7eb-3fd63914be70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" Jul 22 19:23:39.196602 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:23:39.196571 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-serving-cert: secret "maas-api-e2e-ait-cd4f9eba-serving-cert" not found Jul 22 19:23:39.196728 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:23:39.196657 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06d410e9-49c3-41fb-a7eb-3fd63914be70-maas-api-tls podName:06d410e9-49c3-41fb-a7eb-3fd63914be70 nodeName:}" failed. No retries permitted until 2026-07-22 19:23:39.696634701 +0000 UTC m=+1622.608798770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06d410e9-49c3-41fb-a7eb-3fd63914be70-maas-api-tls") pod "maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" (UID: "06d410e9-49c3-41fb-a7eb-3fd63914be70") : secret "maas-api-e2e-ait-cd4f9eba-serving-cert" not found Jul 22 19:23:39.197102 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.197080 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d410e9-49c3-41fb-a7eb-3fd63914be70-openshift-service-ca\") pod \"maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq\" (UID: \"06d410e9-49c3-41fb-a7eb-3fd63914be70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" Jul 22 19:23:39.205135 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.205111 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nkws\" (UniqueName: \"kubernetes.io/projected/06d410e9-49c3-41fb-a7eb-3fd63914be70-kube-api-access-5nkws\") pod \"maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq\" (UID: \"06d410e9-49c3-41fb-a7eb-3fd63914be70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" Jul 22 19:23:39.700753 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.700719 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d410e9-49c3-41fb-a7eb-3fd63914be70-maas-api-tls\") pod \"maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq\" (UID: \"06d410e9-49c3-41fb-a7eb-3fd63914be70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" Jul 22 19:23:39.703392 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.703365 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d410e9-49c3-41fb-a7eb-3fd63914be70-maas-api-tls\") pod \"maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq\" (UID: \"06d410e9-49c3-41fb-a7eb-3fd63914be70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" Jul 22 19:23:39.978013 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:39.977924 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" Jul 22 19:23:40.132689 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:40.132657 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq"] Jul 22 19:23:40.133639 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:23:40.133608 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d410e9_49c3_41fb_a7eb_3fd63914be70.slice/crio-cff6969dace35f443aded16a7b3d5d31d95d17e97c5edcbdeca861adcbe75fba WatchSource:0}: Error finding container cff6969dace35f443aded16a7b3d5d31d95d17e97c5edcbdeca861adcbe75fba: Status 404 returned error can't find the container with id cff6969dace35f443aded16a7b3d5d31d95d17e97c5edcbdeca861adcbe75fba Jul 22 19:23:40.134956 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:40.134938 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 19:23:40.896561 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:40.896521 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" event={"ID":"06d410e9-49c3-41fb-a7eb-3fd63914be70","Type":"ContainerStarted","Data":"369a7bdc9bbc7acc5384a66d3ad33a8398ff5820bdab0f25aa76040378fa48ca"} Jul 22 19:23:40.896763 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:40.896584 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" event={"ID":"06d410e9-49c3-41fb-a7eb-3fd63914be70","Type":"ContainerStarted","Data":"cff6969dace35f443aded16a7b3d5d31d95d17e97c5edcbdeca861adcbe75fba"} Jul 22 19:23:40.896763 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:40.896741 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" Jul 22 19:23:40.913971 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:40.913906 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" podStartSLOduration=1.340056331 podStartE2EDuration="1.913885679s" totalImagesPullingTime="573.829348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:23:40.91220869 +0000 UTC m=+1623.824372954" watchObservedRunningTime="2026-07-22 19:23:40.913885679 +0000 UTC m=+1623.826049747" Jul 22 19:23:46.908157 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:23:46.908128 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" Jul 22 19:24:04.848420 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:04.848379 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2"] Jul 22 19:24:04.852849 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:04.852829 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2" Jul 22 19:24:04.856733 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:04.856701 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jhv4j\"" Jul 22 19:24:04.860474 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:04.860415 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2"] Jul 22 19:24:04.950058 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:04.950016 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e03cc7a-797a-4125-aa35-d013a27556b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2\" (UID: \"4e03cc7a-797a-4125-aa35-d013a27556b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2" Jul 22 19:24:05.051709 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:05.051668 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e03cc7a-797a-4125-aa35-d013a27556b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2\" (UID: \"4e03cc7a-797a-4125-aa35-d013a27556b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2" Jul 22 19:24:05.052308 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:05.052284 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e03cc7a-797a-4125-aa35-d013a27556b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2\" (UID: \"4e03cc7a-797a-4125-aa35-d013a27556b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2" Jul 22 19:24:05.167758 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:05.167667 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2" Jul 22 19:24:05.317550 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:05.317526 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2"] Jul 22 19:24:05.319466 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:24:05.319436 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e03cc7a_797a_4125_aa35_d013a27556b0.slice/crio-897a8820690894b1b35d5429f205d0ff13f554c090714b75f4757fd3c3e99751 WatchSource:0}: Error finding container 897a8820690894b1b35d5429f205d0ff13f554c090714b75f4757fd3c3e99751: Status 404 returned error can't find the container with id 897a8820690894b1b35d5429f205d0ff13f554c090714b75f4757fd3c3e99751 Jul 22 19:24:06.049000 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:06.048966 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2" event={"ID":"4e03cc7a-797a-4125-aa35-d013a27556b0","Type":"ContainerStarted","Data":"577f8d6e6ea10f457d20555ca2a6cba0a137e1944df98084c58f1a6aaaf690c8"} Jul 22 19:24:06.049467 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:06.049025 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2" event={"ID":"4e03cc7a-797a-4125-aa35-d013a27556b0","Type":"ContainerStarted","Data":"897a8820690894b1b35d5429f205d0ff13f554c090714b75f4757fd3c3e99751"} Jul 22 19:24:06.065734 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:06.065687 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2" podStartSLOduration=2.065671554 podStartE2EDuration="2.065671554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:24:06.063663895 +0000 UTC m=+1648.975827965" watchObservedRunningTime="2026-07-22 19:24:06.065671554 +0000 UTC m=+1648.977835622" Jul 22 19:24:07.056793 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:07.056674 2583 generic.go:356] "Generic (PLEG): container finished" podID="4e03cc7a-797a-4125-aa35-d013a27556b0" containerID="577f8d6e6ea10f457d20555ca2a6cba0a137e1944df98084c58f1a6aaaf690c8" exitCode=0 Jul 22 19:24:07.056793 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:07.056748 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2" event={"ID":"4e03cc7a-797a-4125-aa35-d013a27556b0","Type":"ContainerDied","Data":"577f8d6e6ea10f457d20555ca2a6cba0a137e1944df98084c58f1a6aaaf690c8"} Jul 22 19:24:08.198377 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:08.198351 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2" Jul 22 19:24:08.283397 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:08.283360 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e03cc7a-797a-4125-aa35-d013a27556b0-maas-api-service-ca\") pod \"4e03cc7a-797a-4125-aa35-d013a27556b0\" (UID: \"4e03cc7a-797a-4125-aa35-d013a27556b0\") " Jul 22 19:24:08.283804 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:08.283782 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e03cc7a-797a-4125-aa35-d013a27556b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e03cc7a-797a-4125-aa35-d013a27556b0" (UID: "4e03cc7a-797a-4125-aa35-d013a27556b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:24:08.385125 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:08.385037 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e03cc7a-797a-4125-aa35-d013a27556b0-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:24:09.069293 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:09.069261 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2" Jul 22 19:24:09.069514 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:09.069262 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2" event={"ID":"4e03cc7a-797a-4125-aa35-d013a27556b0","Type":"ContainerDied","Data":"897a8820690894b1b35d5429f205d0ff13f554c090714b75f4757fd3c3e99751"} Jul 22 19:24:09.069514 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:09.069389 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="897a8820690894b1b35d5429f205d0ff13f554c090714b75f4757fd3c3e99751" Jul 22 19:24:15.324334 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:15.324294 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq"] Jul 22 19:24:15.324852 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:15.324639 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" podUID="06d410e9-49c3-41fb-a7eb-3fd63914be70" containerName="maas-api" containerID="cri-o://369a7bdc9bbc7acc5384a66d3ad33a8398ff5820bdab0f25aa76040378fa48ca" gracePeriod=30 Jul 22 19:24:15.575625 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:15.575562 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" Jul 22 19:24:15.757840 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:15.757801 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d410e9-49c3-41fb-a7eb-3fd63914be70-openshift-service-ca\") pod \"06d410e9-49c3-41fb-a7eb-3fd63914be70\" (UID: \"06d410e9-49c3-41fb-a7eb-3fd63914be70\") " Jul 22 19:24:15.758030 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:15.757850 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nkws\" (UniqueName: \"kubernetes.io/projected/06d410e9-49c3-41fb-a7eb-3fd63914be70-kube-api-access-5nkws\") pod \"06d410e9-49c3-41fb-a7eb-3fd63914be70\" (UID: \"06d410e9-49c3-41fb-a7eb-3fd63914be70\") " Jul 22 19:24:15.758030 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:15.757871 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d410e9-49c3-41fb-a7eb-3fd63914be70-maas-api-tls\") pod \"06d410e9-49c3-41fb-a7eb-3fd63914be70\" (UID: \"06d410e9-49c3-41fb-a7eb-3fd63914be70\") " Jul 22 19:24:15.758191 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:15.758165 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d410e9-49c3-41fb-a7eb-3fd63914be70-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06d410e9-49c3-41fb-a7eb-3fd63914be70" (UID: "06d410e9-49c3-41fb-a7eb-3fd63914be70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:24:15.760210 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:15.760176 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d410e9-49c3-41fb-a7eb-3fd63914be70-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06d410e9-49c3-41fb-a7eb-3fd63914be70" (UID: "06d410e9-49c3-41fb-a7eb-3fd63914be70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:24:15.760321 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:15.760236 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d410e9-49c3-41fb-a7eb-3fd63914be70-kube-api-access-5nkws" (OuterVolumeSpecName: "kube-api-access-5nkws") pod "06d410e9-49c3-41fb-a7eb-3fd63914be70" (UID: "06d410e9-49c3-41fb-a7eb-3fd63914be70"). InnerVolumeSpecName "kube-api-access-5nkws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:24:15.859623 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:15.859533 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d410e9-49c3-41fb-a7eb-3fd63914be70-openshift-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:24:15.859623 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:15.859563 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nkws\" (UniqueName: \"kubernetes.io/projected/06d410e9-49c3-41fb-a7eb-3fd63914be70-kube-api-access-5nkws\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:24:15.859623 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:15.859574 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d410e9-49c3-41fb-a7eb-3fd63914be70-maas-api-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:24:16.115616 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:16.115528 2583 generic.go:356] "Generic (PLEG): container finished" podID="06d410e9-49c3-41fb-a7eb-3fd63914be70" containerID="369a7bdc9bbc7acc5384a66d3ad33a8398ff5820bdab0f25aa76040378fa48ca" exitCode=0 Jul 22 19:24:16.115616 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:16.115579 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" Jul 22 19:24:16.115811 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:16.115615 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" event={"ID":"06d410e9-49c3-41fb-a7eb-3fd63914be70","Type":"ContainerDied","Data":"369a7bdc9bbc7acc5384a66d3ad33a8398ff5820bdab0f25aa76040378fa48ca"} Jul 22 19:24:16.115811 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:16.115674 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq" event={"ID":"06d410e9-49c3-41fb-a7eb-3fd63914be70","Type":"ContainerDied","Data":"cff6969dace35f443aded16a7b3d5d31d95d17e97c5edcbdeca861adcbe75fba"} Jul 22 19:24:16.115811 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:16.115695 2583 scope.go:117] "RemoveContainer" containerID="369a7bdc9bbc7acc5384a66d3ad33a8398ff5820bdab0f25aa76040378fa48ca" Jul 22 19:24:16.126116 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:16.126097 2583 scope.go:117] "RemoveContainer" containerID="369a7bdc9bbc7acc5384a66d3ad33a8398ff5820bdab0f25aa76040378fa48ca" Jul 22 19:24:16.126417 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:24:16.126398 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"369a7bdc9bbc7acc5384a66d3ad33a8398ff5820bdab0f25aa76040378fa48ca\": container with ID starting with 369a7bdc9bbc7acc5384a66d3ad33a8398ff5820bdab0f25aa76040378fa48ca not found: ID does not exist" containerID="369a7bdc9bbc7acc5384a66d3ad33a8398ff5820bdab0f25aa76040378fa48ca" Jul 22 19:24:16.126483 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:16.126429 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"369a7bdc9bbc7acc5384a66d3ad33a8398ff5820bdab0f25aa76040378fa48ca"} err="failed to get container status \"369a7bdc9bbc7acc5384a66d3ad33a8398ff5820bdab0f25aa76040378fa48ca\": rpc error: code = NotFound desc = could not find container \"369a7bdc9bbc7acc5384a66d3ad33a8398ff5820bdab0f25aa76040378fa48ca\": container with ID starting with 369a7bdc9bbc7acc5384a66d3ad33a8398ff5820bdab0f25aa76040378fa48ca not found: ID does not exist" Jul 22 19:24:16.136932 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:16.136904 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq"] Jul 22 19:24:16.141717 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:16.141686 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd4f9eba-7d77bcb489-qzntq"] Jul 22 19:24:17.551028 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:17.550995 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06d410e9-49c3-41fb-a7eb-3fd63914be70" path="/var/lib/kubelet/pods/06d410e9-49c3-41fb-a7eb-3fd63914be70/volumes" Jul 22 19:24:19.926089 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:19.926044 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2"] Jul 22 19:24:19.928667 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:19.928644 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd4f9eba-w4xg2"] Jul 22 19:24:21.551131 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:21.551088 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e03cc7a-797a-4125-aa35-d013a27556b0" path="/var/lib/kubelet/pods/4e03cc7a-797a-4125-aa35-d013a27556b0/volumes" Jul 22 19:24:23.396482 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.396446 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8"] Jul 22 19:24:23.396917 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.396902 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e03cc7a-797a-4125-aa35-d013a27556b0" containerName="revoke-keys" Jul 22 19:24:23.396917 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.396919 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e03cc7a-797a-4125-aa35-d013a27556b0" containerName="revoke-keys" Jul 22 19:24:23.397008 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.396949 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d410e9-49c3-41fb-a7eb-3fd63914be70" containerName="maas-api" Jul 22 19:24:23.397008 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.396954 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d410e9-49c3-41fb-a7eb-3fd63914be70" containerName="maas-api" Jul 22 19:24:23.397082 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.397022 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="06d410e9-49c3-41fb-a7eb-3fd63914be70" containerName="maas-api" Jul 22 19:24:23.397082 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.397030 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e03cc7a-797a-4125-aa35-d013a27556b0" containerName="revoke-keys" Jul 22 19:24:23.401868 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.401845 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" Jul 22 19:24:23.404363 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.404325 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-6058163f-serving-cert\"" Jul 22 19:24:23.404511 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.404396 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t4d8t\"" Jul 22 19:24:23.412158 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.412108 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8"] Jul 22 19:24:23.425537 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.425482 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xw7g\" (UniqueName: \"kubernetes.io/projected/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-kube-api-access-2xw7g\") pod \"maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8\" (UID: \"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" Jul 22 19:24:23.425717 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.425562 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-maas-api-tls\") pod \"maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8\" (UID: \"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" Jul 22 19:24:23.425717 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.425650 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-openshift-service-ca\") pod \"maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8\" (UID: \"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" Jul 22 19:24:23.526787 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.526740 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-maas-api-tls\") pod \"maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8\" (UID: \"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" Jul 22 19:24:23.526975 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.526863 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-openshift-service-ca\") pod \"maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8\" (UID: \"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" Jul 22 19:24:23.526975 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.526894 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xw7g\" (UniqueName: \"kubernetes.io/projected/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-kube-api-access-2xw7g\") pod \"maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8\" (UID: \"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" Jul 22 19:24:23.527630 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.527602 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-openshift-service-ca\") pod \"maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8\" (UID: \"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" Jul 22 19:24:23.529680 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.529661 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-maas-api-tls\") pod \"maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8\" (UID: \"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" Jul 22 19:24:23.536073 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.536043 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xw7g\" (UniqueName: \"kubernetes.io/projected/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-kube-api-access-2xw7g\") pod \"maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8\" (UID: \"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" Jul 22 19:24:23.717003 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.716902 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" Jul 22 19:24:23.872185 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:23.872153 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8"] Jul 22 19:24:23.873473 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:24:23.873447 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9e8cfb_9ad2_46f4_8e94_839b5487ca46.slice/crio-d4847480e8ee7179d0f310eda2366cfe35b9e60592099a496b74aa1da2bca295 WatchSource:0}: Error finding container d4847480e8ee7179d0f310eda2366cfe35b9e60592099a496b74aa1da2bca295: Status 404 returned error can't find the container with id d4847480e8ee7179d0f310eda2366cfe35b9e60592099a496b74aa1da2bca295 Jul 22 19:24:24.167512 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:24.167468 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" event={"ID":"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46","Type":"ContainerStarted","Data":"d4847480e8ee7179d0f310eda2366cfe35b9e60592099a496b74aa1da2bca295"} Jul 22 19:24:25.175423 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:25.175388 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" event={"ID":"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46","Type":"ContainerStarted","Data":"345b735f7ac6ac19809c33454662a74f0596b38b09971fc5e61d062cc26fcf54"} Jul 22 19:24:25.175902 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:25.175527 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" Jul 22 19:24:25.191337 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:25.191286 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" podStartSLOduration=1.7225088720000001 podStartE2EDuration="2.191268961s" totalImagesPullingTime="468.760089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:24:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:24:25.19116347 +0000 UTC m=+1668.103327551" watchObservedRunningTime="2026-07-22 19:24:25.191268961 +0000 UTC m=+1668.103433032" Jul 22 19:24:31.187411 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:31.187384 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" Jul 22 19:24:48.477943 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:48.477904 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6"] Jul 22 19:24:48.482710 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:48.482683 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6" Jul 22 19:24:48.486241 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:48.486101 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jhv4j\"" Jul 22 19:24:48.489393 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:48.489367 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6"] Jul 22 19:24:48.582694 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:48.582661 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f91cdad-9992-453c-b938-497ab7f8f8cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6058163f-smhp6\" (UID: \"2f91cdad-9992-453c-b938-497ab7f8f8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6" Jul 22 19:24:48.683507 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:48.683472 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f91cdad-9992-453c-b938-497ab7f8f8cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6058163f-smhp6\" (UID: \"2f91cdad-9992-453c-b938-497ab7f8f8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6" Jul 22 19:24:48.684148 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:48.684121 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f91cdad-9992-453c-b938-497ab7f8f8cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6058163f-smhp6\" (UID: \"2f91cdad-9992-453c-b938-497ab7f8f8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6" Jul 22 19:24:48.795627 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:48.795590 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6" Jul 22 19:24:48.956370 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:48.956342 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6"] Jul 22 19:24:49.330244 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:49.330202 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6" event={"ID":"2f91cdad-9992-453c-b938-497ab7f8f8cb","Type":"ContainerStarted","Data":"244e203b37f92f78893714d02718ee5ce72312057123e633cf53c60ccb170402"} Jul 22 19:24:49.330463 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:49.330308 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6" event={"ID":"2f91cdad-9992-453c-b938-497ab7f8f8cb","Type":"ContainerStarted","Data":"11cda34ee2159e2d49d800a51032b6438bcb2d798dcbaff30face0987cc59c45"} Jul 22 19:24:49.343927 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:49.343867 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6" podStartSLOduration=1.343852873 podStartE2EDuration="1.343852873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:24:49.343589388 +0000 UTC m=+1692.255753458" watchObservedRunningTime="2026-07-22 19:24:49.343852873 +0000 UTC m=+1692.256016940" Jul 22 19:24:50.337675 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:50.337634 2583 generic.go:356] "Generic (PLEG): container finished" podID="2f91cdad-9992-453c-b938-497ab7f8f8cb" containerID="244e203b37f92f78893714d02718ee5ce72312057123e633cf53c60ccb170402" exitCode=0 Jul 22 19:24:50.338072 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:50.337723 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6" event={"ID":"2f91cdad-9992-453c-b938-497ab7f8f8cb","Type":"ContainerDied","Data":"244e203b37f92f78893714d02718ee5ce72312057123e633cf53c60ccb170402"} Jul 22 19:24:51.482895 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:51.482871 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6" Jul 22 19:24:51.613293 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:51.613198 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f91cdad-9992-453c-b938-497ab7f8f8cb-maas-api-service-ca\") pod \"2f91cdad-9992-453c-b938-497ab7f8f8cb\" (UID: \"2f91cdad-9992-453c-b938-497ab7f8f8cb\") " Jul 22 19:24:51.613558 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:51.613533 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f91cdad-9992-453c-b938-497ab7f8f8cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2f91cdad-9992-453c-b938-497ab7f8f8cb" (UID: "2f91cdad-9992-453c-b938-497ab7f8f8cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:24:51.714786 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:51.714750 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f91cdad-9992-453c-b938-497ab7f8f8cb-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:24:52.351136 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:52.351104 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6" Jul 22 19:24:52.351314 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:52.351137 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6" event={"ID":"2f91cdad-9992-453c-b938-497ab7f8f8cb","Type":"ContainerDied","Data":"11cda34ee2159e2d49d800a51032b6438bcb2d798dcbaff30face0987cc59c45"} Jul 22 19:24:52.351314 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:52.351186 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11cda34ee2159e2d49d800a51032b6438bcb2d798dcbaff30face0987cc59c45" Jul 22 19:24:59.020178 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.020080 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8"] Jul 22 19:24:59.020643 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.020317 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" podUID="4d9e8cfb-9ad2-46f4-8e94-839b5487ca46" containerName="maas-api" containerID="cri-o://345b735f7ac6ac19809c33454662a74f0596b38b09971fc5e61d062cc26fcf54" gracePeriod=30 Jul 22 19:24:59.278929 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.278863 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" Jul 22 19:24:59.384029 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.383986 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-maas-api-tls\") pod \"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46\" (UID: \"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46\") " Jul 22 19:24:59.384235 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.384111 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xw7g\" (UniqueName: \"kubernetes.io/projected/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-kube-api-access-2xw7g\") pod \"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46\" (UID: \"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46\") " Jul 22 19:24:59.384235 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.384140 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-openshift-service-ca\") pod \"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46\" (UID: \"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46\") " Jul 22 19:24:59.384627 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.384562 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d9e8cfb-9ad2-46f4-8e94-839b5487ca46" (UID: "4d9e8cfb-9ad2-46f4-8e94-839b5487ca46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:24:59.386703 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.386673 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d9e8cfb-9ad2-46f4-8e94-839b5487ca46" (UID: "4d9e8cfb-9ad2-46f4-8e94-839b5487ca46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:24:59.387700 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.387664 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-kube-api-access-2xw7g" (OuterVolumeSpecName: "kube-api-access-2xw7g") pod "4d9e8cfb-9ad2-46f4-8e94-839b5487ca46" (UID: "4d9e8cfb-9ad2-46f4-8e94-839b5487ca46"). InnerVolumeSpecName "kube-api-access-2xw7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:24:59.395909 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.395877 2583 generic.go:356] "Generic (PLEG): container finished" podID="4d9e8cfb-9ad2-46f4-8e94-839b5487ca46" containerID="345b735f7ac6ac19809c33454662a74f0596b38b09971fc5e61d062cc26fcf54" exitCode=0 Jul 22 19:24:59.396051 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.395913 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" Jul 22 19:24:59.396051 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.396004 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" event={"ID":"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46","Type":"ContainerDied","Data":"345b735f7ac6ac19809c33454662a74f0596b38b09971fc5e61d062cc26fcf54"} Jul 22 19:24:59.396203 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.396066 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8" event={"ID":"4d9e8cfb-9ad2-46f4-8e94-839b5487ca46","Type":"ContainerDied","Data":"d4847480e8ee7179d0f310eda2366cfe35b9e60592099a496b74aa1da2bca295"} Jul 22 19:24:59.396203 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.396099 2583 scope.go:117] "RemoveContainer" containerID="345b735f7ac6ac19809c33454662a74f0596b38b09971fc5e61d062cc26fcf54" Jul 22 19:24:59.412646 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.412620 2583 scope.go:117] "RemoveContainer" containerID="345b735f7ac6ac19809c33454662a74f0596b38b09971fc5e61d062cc26fcf54" Jul 22 19:24:59.413032 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:24:59.413007 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345b735f7ac6ac19809c33454662a74f0596b38b09971fc5e61d062cc26fcf54\": container with ID starting with 345b735f7ac6ac19809c33454662a74f0596b38b09971fc5e61d062cc26fcf54 not found: ID does not exist" containerID="345b735f7ac6ac19809c33454662a74f0596b38b09971fc5e61d062cc26fcf54" Jul 22 19:24:59.413153 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.413042 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345b735f7ac6ac19809c33454662a74f0596b38b09971fc5e61d062cc26fcf54"} err="failed to get container status \"345b735f7ac6ac19809c33454662a74f0596b38b09971fc5e61d062cc26fcf54\": rpc error: code = NotFound desc = could not find container \"345b735f7ac6ac19809c33454662a74f0596b38b09971fc5e61d062cc26fcf54\": container with ID starting with 345b735f7ac6ac19809c33454662a74f0596b38b09971fc5e61d062cc26fcf54 not found: ID does not exist" Jul 22 19:24:59.432152 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.432105 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8"] Jul 22 19:24:59.434572 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.434548 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6058163f-746c84f5f5-6vmg8"] Jul 22 19:24:59.485431 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.485391 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-maas-api-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:24:59.485431 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.485431 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xw7g\" (UniqueName: \"kubernetes.io/projected/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-kube-api-access-2xw7g\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:24:59.485650 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.485446 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46-openshift-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:24:59.551243 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:24:59.551160 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d9e8cfb-9ad2-46f4-8e94-839b5487ca46" path="/var/lib/kubelet/pods/4d9e8cfb-9ad2-46f4-8e94-839b5487ca46/volumes" Jul 22 19:25:03.551706 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:03.551673 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6"] Jul 22 19:25:03.553068 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:03.553048 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6058163f-smhp6"] Jul 22 19:25:05.551154 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:05.551113 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f91cdad-9992-453c-b938-497ab7f8f8cb" path="/var/lib/kubelet/pods/2f91cdad-9992-453c-b938-497ab7f8f8cb/volumes" Jul 22 19:25:07.472471 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.472434 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j"] Jul 22 19:25:07.473149 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.473131 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9e8cfb-9ad2-46f4-8e94-839b5487ca46" containerName="maas-api" Jul 22 19:25:07.473208 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.473154 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9e8cfb-9ad2-46f4-8e94-839b5487ca46" containerName="maas-api" Jul 22 19:25:07.473208 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.473169 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f91cdad-9992-453c-b938-497ab7f8f8cb" containerName="revoke-keys" Jul 22 19:25:07.473208 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.473178 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f91cdad-9992-453c-b938-497ab7f8f8cb" containerName="revoke-keys" Jul 22 19:25:07.473332 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.473319 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f91cdad-9992-453c-b938-497ab7f8f8cb" containerName="revoke-keys" Jul 22 19:25:07.473366 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.473345 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9e8cfb-9ad2-46f4-8e94-839b5487ca46" containerName="maas-api" Jul 22 19:25:07.476622 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.476606 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:07.479084 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.479033 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9c70fc9e-serving-cert\"" Jul 22 19:25:07.479253 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.479212 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t4d8t\"" Jul 22 19:25:07.488226 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.488197 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j"] Jul 22 19:25:07.568803 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.568768 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db66711-fa03-495f-820d-f609158e3af5-maas-api-tls\") pod \"maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j\" (UID: \"6db66711-fa03-495f-820d-f609158e3af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:07.568965 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.568858 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db66711-fa03-495f-820d-f609158e3af5-openshift-service-ca\") pod \"maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j\" (UID: \"6db66711-fa03-495f-820d-f609158e3af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:07.568965 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.568882 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt5vq\" (UniqueName: \"kubernetes.io/projected/6db66711-fa03-495f-820d-f609158e3af5-kube-api-access-jt5vq\") pod \"maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j\" (UID: \"6db66711-fa03-495f-820d-f609158e3af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:07.670000 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.669962 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db66711-fa03-495f-820d-f609158e3af5-openshift-service-ca\") pod \"maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j\" (UID: \"6db66711-fa03-495f-820d-f609158e3af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:07.670202 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.670007 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt5vq\" (UniqueName: \"kubernetes.io/projected/6db66711-fa03-495f-820d-f609158e3af5-kube-api-access-jt5vq\") pod \"maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j\" (UID: \"6db66711-fa03-495f-820d-f609158e3af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:07.670202 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.670100 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db66711-fa03-495f-820d-f609158e3af5-maas-api-tls\") pod \"maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j\" (UID: \"6db66711-fa03-495f-820d-f609158e3af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:07.670294 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:25:07.670251 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-serving-cert: secret "maas-api-e2e-ait-9c70fc9e-serving-cert" not found Jul 22 19:25:07.670336 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:25:07.670313 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6db66711-fa03-495f-820d-f609158e3af5-maas-api-tls podName:6db66711-fa03-495f-820d-f609158e3af5 nodeName:}" failed. No retries permitted until 2026-07-22 19:25:08.170296671 +0000 UTC m=+1711.082460718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6db66711-fa03-495f-820d-f609158e3af5-maas-api-tls") pod "maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" (UID: "6db66711-fa03-495f-820d-f609158e3af5") : secret "maas-api-e2e-ait-9c70fc9e-serving-cert" not found Jul 22 19:25:07.670625 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.670605 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db66711-fa03-495f-820d-f609158e3af5-openshift-service-ca\") pod \"maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j\" (UID: \"6db66711-fa03-495f-820d-f609158e3af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:07.681060 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:07.681030 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt5vq\" (UniqueName: \"kubernetes.io/projected/6db66711-fa03-495f-820d-f609158e3af5-kube-api-access-jt5vq\") pod \"maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j\" (UID: \"6db66711-fa03-495f-820d-f609158e3af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:08.174735 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:08.174697 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db66711-fa03-495f-820d-f609158e3af5-maas-api-tls\") pod \"maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j\" (UID: \"6db66711-fa03-495f-820d-f609158e3af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:08.174924 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:25:08.174851 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-serving-cert: secret "maas-api-e2e-ait-9c70fc9e-serving-cert" not found Jul 22 19:25:08.174969 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:25:08.174924 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6db66711-fa03-495f-820d-f609158e3af5-maas-api-tls podName:6db66711-fa03-495f-820d-f609158e3af5 nodeName:}" failed. No retries permitted until 2026-07-22 19:25:09.174907529 +0000 UTC m=+1712.087071596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6db66711-fa03-495f-820d-f609158e3af5-maas-api-tls") pod "maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" (UID: "6db66711-fa03-495f-820d-f609158e3af5") : secret "maas-api-e2e-ait-9c70fc9e-serving-cert" not found Jul 22 19:25:09.185046 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:09.185012 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db66711-fa03-495f-820d-f609158e3af5-maas-api-tls\") pod \"maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j\" (UID: \"6db66711-fa03-495f-820d-f609158e3af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:09.187697 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:09.187676 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db66711-fa03-495f-820d-f609158e3af5-maas-api-tls\") pod \"maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j\" (UID: \"6db66711-fa03-495f-820d-f609158e3af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:09.293079 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:09.293039 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:09.440090 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:09.440061 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j"] Jul 22 19:25:09.441350 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:25:09.441323 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db66711_fa03_495f_820d_f609158e3af5.slice/crio-77ab882082bf88d37daefff7a55ac11f623e1d0aa34207b2a696d291251cf70c WatchSource:0}: Error finding container 77ab882082bf88d37daefff7a55ac11f623e1d0aa34207b2a696d291251cf70c: Status 404 returned error can't find the container with id 77ab882082bf88d37daefff7a55ac11f623e1d0aa34207b2a696d291251cf70c Jul 22 19:25:09.459624 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:09.459596 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" event={"ID":"6db66711-fa03-495f-820d-f609158e3af5","Type":"ContainerStarted","Data":"77ab882082bf88d37daefff7a55ac11f623e1d0aa34207b2a696d291251cf70c"} Jul 22 19:25:10.467099 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:10.467052 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" event={"ID":"6db66711-fa03-495f-820d-f609158e3af5","Type":"ContainerStarted","Data":"d6d6d104c9a400450e1485b7f5b4a7e5e1677c4cff5f3dc54fd9bbf589a15773"} Jul 22 19:25:10.467526 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:10.467210 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:10.485539 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:10.485471 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" podStartSLOduration=3.1428175 podStartE2EDuration="3.485457078s" totalImagesPullingTime="342.639578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:25:10.483900849 +0000 UTC m=+1713.396064909" watchObservedRunningTime="2026-07-22 19:25:10.485457078 +0000 UTC m=+1713.397621145" Jul 22 19:25:16.478764 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:16.478733 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:33.568128 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:33.568091 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx"] Jul 22 19:25:33.572125 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:33.572101 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx" Jul 22 19:25:33.575526 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:33.575477 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jhv4j\"" Jul 22 19:25:33.578788 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:33.578764 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx"] Jul 22 19:25:33.623177 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:33.623134 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f697199-6672-45ad-a01d-0f70ea7fcee7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx\" (UID: \"6f697199-6672-45ad-a01d-0f70ea7fcee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx" Jul 22 19:25:33.724424 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:33.724390 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f697199-6672-45ad-a01d-0f70ea7fcee7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx\" (UID: \"6f697199-6672-45ad-a01d-0f70ea7fcee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx" Jul 22 19:25:33.725037 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:33.725016 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f697199-6672-45ad-a01d-0f70ea7fcee7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx\" (UID: \"6f697199-6672-45ad-a01d-0f70ea7fcee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx" Jul 22 19:25:33.884970 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:33.884879 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx" Jul 22 19:25:34.032986 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:34.032946 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx"] Jul 22 19:25:34.033334 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:25:34.033312 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f697199_6672_45ad_a01d_0f70ea7fcee7.slice/crio-e538597fb2d9fab303d316e7b08edbb6ab4c259690e760a36fef74c3eb5c1c9e WatchSource:0}: Error finding container e538597fb2d9fab303d316e7b08edbb6ab4c259690e760a36fef74c3eb5c1c9e: Status 404 returned error can't find the container with id e538597fb2d9fab303d316e7b08edbb6ab4c259690e760a36fef74c3eb5c1c9e Jul 22 19:25:34.606163 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:34.606124 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx" event={"ID":"6f697199-6672-45ad-a01d-0f70ea7fcee7","Type":"ContainerStarted","Data":"8252d56aeefd2e816ad628c8d34712fccf3eeedf0edb7b4483011767d56f407e"} Jul 22 19:25:34.606642 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:34.606178 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx" event={"ID":"6f697199-6672-45ad-a01d-0f70ea7fcee7","Type":"ContainerStarted","Data":"e538597fb2d9fab303d316e7b08edbb6ab4c259690e760a36fef74c3eb5c1c9e"} Jul 22 19:25:34.622212 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:34.622162 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx" podStartSLOduration=1.6221480320000001 podStartE2EDuration="1.622148032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:25:34.620974761 +0000 UTC m=+1737.533138830" watchObservedRunningTime="2026-07-22 19:25:34.622148032 +0000 UTC m=+1737.534312137" Jul 22 19:25:35.615411 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:35.615376 2583 generic.go:356] "Generic (PLEG): container finished" podID="6f697199-6672-45ad-a01d-0f70ea7fcee7" containerID="8252d56aeefd2e816ad628c8d34712fccf3eeedf0edb7b4483011767d56f407e" exitCode=0 Jul 22 19:25:35.615834 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:35.615424 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx" event={"ID":"6f697199-6672-45ad-a01d-0f70ea7fcee7","Type":"ContainerDied","Data":"8252d56aeefd2e816ad628c8d34712fccf3eeedf0edb7b4483011767d56f407e"} Jul 22 19:25:36.757305 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:36.757280 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx" Jul 22 19:25:36.851364 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:36.851331 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f697199-6672-45ad-a01d-0f70ea7fcee7-maas-api-service-ca\") pod \"6f697199-6672-45ad-a01d-0f70ea7fcee7\" (UID: \"6f697199-6672-45ad-a01d-0f70ea7fcee7\") " Jul 22 19:25:36.851759 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:36.851736 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f697199-6672-45ad-a01d-0f70ea7fcee7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f697199-6672-45ad-a01d-0f70ea7fcee7" (UID: "6f697199-6672-45ad-a01d-0f70ea7fcee7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:25:36.952436 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:36.952350 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f697199-6672-45ad-a01d-0f70ea7fcee7-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:25:37.629400 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:37.629370 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx" Jul 22 19:25:37.629400 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:37.629385 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx" event={"ID":"6f697199-6672-45ad-a01d-0f70ea7fcee7","Type":"ContainerDied","Data":"e538597fb2d9fab303d316e7b08edbb6ab4c259690e760a36fef74c3eb5c1c9e"} Jul 22 19:25:37.629645 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:37.629427 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e538597fb2d9fab303d316e7b08edbb6ab4c259690e760a36fef74c3eb5c1c9e" Jul 22 19:25:43.964881 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:43.964822 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j"] Jul 22 19:25:43.965381 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:43.965093 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" podUID="6db66711-fa03-495f-820d-f609158e3af5" containerName="maas-api" containerID="cri-o://d6d6d104c9a400450e1485b7f5b4a7e5e1677c4cff5f3dc54fd9bbf589a15773" gracePeriod=30 Jul 22 19:25:44.221186 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.221124 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:44.325519 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.325453 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db66711-fa03-495f-820d-f609158e3af5-openshift-service-ca\") pod \"6db66711-fa03-495f-820d-f609158e3af5\" (UID: \"6db66711-fa03-495f-820d-f609158e3af5\") " Jul 22 19:25:44.325519 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.325515 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jt5vq\" (UniqueName: \"kubernetes.io/projected/6db66711-fa03-495f-820d-f609158e3af5-kube-api-access-jt5vq\") pod \"6db66711-fa03-495f-820d-f609158e3af5\" (UID: \"6db66711-fa03-495f-820d-f609158e3af5\") " Jul 22 19:25:44.325751 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.325647 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db66711-fa03-495f-820d-f609158e3af5-maas-api-tls\") pod \"6db66711-fa03-495f-820d-f609158e3af5\" (UID: \"6db66711-fa03-495f-820d-f609158e3af5\") " Jul 22 19:25:44.325815 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.325792 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db66711-fa03-495f-820d-f609158e3af5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6db66711-fa03-495f-820d-f609158e3af5" (UID: "6db66711-fa03-495f-820d-f609158e3af5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:25:44.327778 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.327749 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db66711-fa03-495f-820d-f609158e3af5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6db66711-fa03-495f-820d-f609158e3af5" (UID: "6db66711-fa03-495f-820d-f609158e3af5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:25:44.327875 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.327780 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db66711-fa03-495f-820d-f609158e3af5-kube-api-access-jt5vq" (OuterVolumeSpecName: "kube-api-access-jt5vq") pod "6db66711-fa03-495f-820d-f609158e3af5" (UID: "6db66711-fa03-495f-820d-f609158e3af5"). InnerVolumeSpecName "kube-api-access-jt5vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:25:44.427078 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.427035 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db66711-fa03-495f-820d-f609158e3af5-openshift-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:25:44.427078 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.427068 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt5vq\" (UniqueName: \"kubernetes.io/projected/6db66711-fa03-495f-820d-f609158e3af5-kube-api-access-jt5vq\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:25:44.427078 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.427079 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db66711-fa03-495f-820d-f609158e3af5-maas-api-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:25:44.670247 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.670214 2583 generic.go:356] "Generic (PLEG): container finished" podID="6db66711-fa03-495f-820d-f609158e3af5" containerID="d6d6d104c9a400450e1485b7f5b4a7e5e1677c4cff5f3dc54fd9bbf589a15773" exitCode=0 Jul 22 19:25:44.670449 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.670258 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" Jul 22 19:25:44.670449 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.670313 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" event={"ID":"6db66711-fa03-495f-820d-f609158e3af5","Type":"ContainerDied","Data":"d6d6d104c9a400450e1485b7f5b4a7e5e1677c4cff5f3dc54fd9bbf589a15773"} Jul 22 19:25:44.670449 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.670371 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j" event={"ID":"6db66711-fa03-495f-820d-f609158e3af5","Type":"ContainerDied","Data":"77ab882082bf88d37daefff7a55ac11f623e1d0aa34207b2a696d291251cf70c"} Jul 22 19:25:44.670449 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.670399 2583 scope.go:117] "RemoveContainer" containerID="d6d6d104c9a400450e1485b7f5b4a7e5e1677c4cff5f3dc54fd9bbf589a15773" Jul 22 19:25:44.680719 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.680698 2583 scope.go:117] "RemoveContainer" containerID="d6d6d104c9a400450e1485b7f5b4a7e5e1677c4cff5f3dc54fd9bbf589a15773" Jul 22 19:25:44.681019 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:25:44.680996 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d6d104c9a400450e1485b7f5b4a7e5e1677c4cff5f3dc54fd9bbf589a15773\": container with ID starting with d6d6d104c9a400450e1485b7f5b4a7e5e1677c4cff5f3dc54fd9bbf589a15773 not found: ID does not exist" containerID="d6d6d104c9a400450e1485b7f5b4a7e5e1677c4cff5f3dc54fd9bbf589a15773" Jul 22 19:25:44.681070 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.681028 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d6d104c9a400450e1485b7f5b4a7e5e1677c4cff5f3dc54fd9bbf589a15773"} err="failed to get container status \"d6d6d104c9a400450e1485b7f5b4a7e5e1677c4cff5f3dc54fd9bbf589a15773\": rpc error: code = NotFound desc = could not find container \"d6d6d104c9a400450e1485b7f5b4a7e5e1677c4cff5f3dc54fd9bbf589a15773\": container with ID starting with d6d6d104c9a400450e1485b7f5b4a7e5e1677c4cff5f3dc54fd9bbf589a15773 not found: ID does not exist" Jul 22 19:25:44.692365 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.692333 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j"] Jul 22 19:25:44.696435 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:44.696410 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c70fc9e-74cddb77cc-6xt5j"] Jul 22 19:25:45.551520 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:45.551472 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6db66711-fa03-495f-820d-f609158e3af5" path="/var/lib/kubelet/pods/6db66711-fa03-495f-820d-f609158e3af5/volumes" Jul 22 19:25:48.643645 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:48.643615 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx"] Jul 22 19:25:48.646330 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:48.646301 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c70fc9e-mzslx"] Jul 22 19:25:49.551990 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:49.551952 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f697199-6672-45ad-a01d-0f70ea7fcee7" path="/var/lib/kubelet/pods/6f697199-6672-45ad-a01d-0f70ea7fcee7/volumes" Jul 22 19:25:53.857437 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:53.857405 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh"] Jul 22 19:25:53.858106 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:53.858085 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db66711-fa03-495f-820d-f609158e3af5" containerName="maas-api" Jul 22 19:25:53.858106 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:53.858107 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db66711-fa03-495f-820d-f609158e3af5" containerName="maas-api" Jul 22 19:25:53.858272 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:53.858124 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f697199-6672-45ad-a01d-0f70ea7fcee7" containerName="revoke-keys" Jul 22 19:25:53.858272 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:53.858151 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f697199-6672-45ad-a01d-0f70ea7fcee7" containerName="revoke-keys" Jul 22 19:25:53.858386 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:53.858345 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db66711-fa03-495f-820d-f609158e3af5" containerName="maas-api" Jul 22 19:25:53.858386 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:53.858368 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f697199-6672-45ad-a01d-0f70ea7fcee7" containerName="revoke-keys" Jul 22 19:25:53.863007 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:53.862984 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" Jul 22 19:25:53.865360 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:53.865334 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-eaab3c34-serving-cert\"" Jul 22 19:25:53.866473 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:53.866454 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t4d8t\"" Jul 22 19:25:53.870074 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:53.870052 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh"] Jul 22 19:25:53.924157 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:53.924113 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6847015-1c4e-4c80-a134-df0a6fc05519-maas-api-tls\") pod \"maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh\" (UID: \"c6847015-1c4e-4c80-a134-df0a6fc05519\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" Jul 22 19:25:53.924157 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:53.924161 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqqmr\" (UniqueName: \"kubernetes.io/projected/c6847015-1c4e-4c80-a134-df0a6fc05519-kube-api-access-mqqmr\") pod \"maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh\" (UID: \"c6847015-1c4e-4c80-a134-df0a6fc05519\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" Jul 22 19:25:53.924393 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:53.924201 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6847015-1c4e-4c80-a134-df0a6fc05519-openshift-service-ca\") pod \"maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh\" (UID: \"c6847015-1c4e-4c80-a134-df0a6fc05519\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" Jul 22 19:25:54.025021 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:54.024985 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6847015-1c4e-4c80-a134-df0a6fc05519-maas-api-tls\") pod \"maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh\" (UID: \"c6847015-1c4e-4c80-a134-df0a6fc05519\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" Jul 22 19:25:54.025206 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:54.025039 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqqmr\" (UniqueName: \"kubernetes.io/projected/c6847015-1c4e-4c80-a134-df0a6fc05519-kube-api-access-mqqmr\") pod \"maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh\" (UID: \"c6847015-1c4e-4c80-a134-df0a6fc05519\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" Jul 22 19:25:54.025206 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:54.025066 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6847015-1c4e-4c80-a134-df0a6fc05519-openshift-service-ca\") pod \"maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh\" (UID: \"c6847015-1c4e-4c80-a134-df0a6fc05519\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" Jul 22 19:25:54.025206 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:25:54.025139 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-serving-cert: secret "maas-api-e2e-derive-eaab3c34-serving-cert" not found Jul 22 19:25:54.025206 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:25:54.025205 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6847015-1c4e-4c80-a134-df0a6fc05519-maas-api-tls podName:c6847015-1c4e-4c80-a134-df0a6fc05519 nodeName:}" failed. No retries permitted until 2026-07-22 19:25:54.525179184 +0000 UTC m=+1757.437343230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c6847015-1c4e-4c80-a134-df0a6fc05519-maas-api-tls") pod "maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" (UID: "c6847015-1c4e-4c80-a134-df0a6fc05519") : secret "maas-api-e2e-derive-eaab3c34-serving-cert" not found Jul 22 19:25:54.025799 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:54.025774 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6847015-1c4e-4c80-a134-df0a6fc05519-openshift-service-ca\") pod \"maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh\" (UID: \"c6847015-1c4e-4c80-a134-df0a6fc05519\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" Jul 22 19:25:54.035871 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:54.035844 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqqmr\" (UniqueName: \"kubernetes.io/projected/c6847015-1c4e-4c80-a134-df0a6fc05519-kube-api-access-mqqmr\") pod \"maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh\" (UID: \"c6847015-1c4e-4c80-a134-df0a6fc05519\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" Jul 22 19:25:54.530974 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:54.530925 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6847015-1c4e-4c80-a134-df0a6fc05519-maas-api-tls\") pod \"maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh\" (UID: \"c6847015-1c4e-4c80-a134-df0a6fc05519\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" Jul 22 19:25:54.533652 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:54.533625 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6847015-1c4e-4c80-a134-df0a6fc05519-maas-api-tls\") pod \"maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh\" (UID: \"c6847015-1c4e-4c80-a134-df0a6fc05519\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" Jul 22 19:25:54.775997 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:54.775965 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" Jul 22 19:25:54.921804 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:54.921774 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh"] Jul 22 19:25:54.923441 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:25:54.923403 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6847015_1c4e_4c80_a134_df0a6fc05519.slice/crio-fdf8c26857d059281d7a34fb9eb9ac6b7e59460243e53b40d6d5b3575700d572 WatchSource:0}: Error finding container fdf8c26857d059281d7a34fb9eb9ac6b7e59460243e53b40d6d5b3575700d572: Status 404 returned error can't find the container with id fdf8c26857d059281d7a34fb9eb9ac6b7e59460243e53b40d6d5b3575700d572 Jul 22 19:25:55.738219 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:55.738129 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" event={"ID":"c6847015-1c4e-4c80-a134-df0a6fc05519","Type":"ContainerStarted","Data":"4483b1288a8c0017363b8ee0461cc523f99f897b4d7861def7c49a959777a31e"} Jul 22 19:25:55.738219 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:55.738190 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" event={"ID":"c6847015-1c4e-4c80-a134-df0a6fc05519","Type":"ContainerStarted","Data":"fdf8c26857d059281d7a34fb9eb9ac6b7e59460243e53b40d6d5b3575700d572"} Jul 22 19:25:55.738412 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:55.738328 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" Jul 22 19:25:55.755407 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:25:55.755351 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" podStartSLOduration=2.305363799 podStartE2EDuration="2.755333021s" totalImagesPullingTime="449.969222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:25:55.753652628 +0000 UTC m=+1758.665816695" watchObservedRunningTime="2026-07-22 19:25:55.755333021 +0000 UTC m=+1758.667497089" Jul 22 19:26:01.767103 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:01.767072 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" Jul 22 19:26:18.628786 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:18.628751 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx"] Jul 22 19:26:18.632421 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:18.632403 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx" Jul 22 19:26:18.634998 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:18.634970 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jhv4j\"" Jul 22 19:26:18.640770 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:18.640675 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx"] Jul 22 19:26:18.650640 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:18.650610 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee9e49db-726c-45a1-a068-f674e2137d87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx\" (UID: \"ee9e49db-726c-45a1-a068-f674e2137d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx" Jul 22 19:26:18.751791 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:18.751755 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee9e49db-726c-45a1-a068-f674e2137d87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx\" (UID: \"ee9e49db-726c-45a1-a068-f674e2137d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx" Jul 22 19:26:18.752415 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:18.752391 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee9e49db-726c-45a1-a068-f674e2137d87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx\" (UID: \"ee9e49db-726c-45a1-a068-f674e2137d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx" Jul 22 19:26:18.944698 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:18.944614 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx" Jul 22 19:26:19.090746 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:26:19.090707 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9e49db_726c_45a1_a068_f674e2137d87.slice/crio-6d05bc3f5d7455034dd417698264c7ed055122ade02ae4524fb6714b1a181e99 WatchSource:0}: Error finding container 6d05bc3f5d7455034dd417698264c7ed055122ade02ae4524fb6714b1a181e99: Status 404 returned error can't find the container with id 6d05bc3f5d7455034dd417698264c7ed055122ade02ae4524fb6714b1a181e99 Jul 22 19:26:19.091261 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:19.091213 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx"] Jul 22 19:26:19.922077 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:19.922040 2583 generic.go:356] "Generic (PLEG): container finished" podID="ee9e49db-726c-45a1-a068-f674e2137d87" containerID="3f83e25122fcc073cda6f7d35df5ebad41626690c98821cb456878fc3bfc2f55" exitCode=0 Jul 22 19:26:19.922483 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:19.922125 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx" event={"ID":"ee9e49db-726c-45a1-a068-f674e2137d87","Type":"ContainerDied","Data":"3f83e25122fcc073cda6f7d35df5ebad41626690c98821cb456878fc3bfc2f55"} Jul 22 19:26:19.922483 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:19.922178 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx" event={"ID":"ee9e49db-726c-45a1-a068-f674e2137d87","Type":"ContainerStarted","Data":"6d05bc3f5d7455034dd417698264c7ed055122ade02ae4524fb6714b1a181e99"} Jul 22 19:26:21.069407 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:21.069381 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx" Jul 22 19:26:21.173514 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:21.173461 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee9e49db-726c-45a1-a068-f674e2137d87-maas-api-service-ca\") pod \"ee9e49db-726c-45a1-a068-f674e2137d87\" (UID: \"ee9e49db-726c-45a1-a068-f674e2137d87\") " Jul 22 19:26:21.173830 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:21.173808 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee9e49db-726c-45a1-a068-f674e2137d87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ee9e49db-726c-45a1-a068-f674e2137d87" (UID: "ee9e49db-726c-45a1-a068-f674e2137d87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:26:21.275101 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:21.275071 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee9e49db-726c-45a1-a068-f674e2137d87-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:26:21.935468 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:21.935384 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx" Jul 22 19:26:21.935468 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:21.935435 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx" event={"ID":"ee9e49db-726c-45a1-a068-f674e2137d87","Type":"ContainerDied","Data":"6d05bc3f5d7455034dd417698264c7ed055122ade02ae4524fb6714b1a181e99"} Jul 22 19:26:21.935682 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:21.935481 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d05bc3f5d7455034dd417698264c7ed055122ade02ae4524fb6714b1a181e99" Jul 22 19:26:29.005209 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.005103 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh"] Jul 22 19:26:29.005698 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.005557 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" podUID="c6847015-1c4e-4c80-a134-df0a6fc05519" containerName="maas-api" containerID="cri-o://4483b1288a8c0017363b8ee0461cc523f99f897b4d7861def7c49a959777a31e" gracePeriod=30 Jul 22 19:26:29.261580 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.261520 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" Jul 22 19:26:29.359865 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.359826 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6847015-1c4e-4c80-a134-df0a6fc05519-openshift-service-ca\") pod \"c6847015-1c4e-4c80-a134-df0a6fc05519\" (UID: \"c6847015-1c4e-4c80-a134-df0a6fc05519\") " Jul 22 19:26:29.360059 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.359884 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqqmr\" (UniqueName: \"kubernetes.io/projected/c6847015-1c4e-4c80-a134-df0a6fc05519-kube-api-access-mqqmr\") pod \"c6847015-1c4e-4c80-a134-df0a6fc05519\" (UID: \"c6847015-1c4e-4c80-a134-df0a6fc05519\") " Jul 22 19:26:29.360059 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.359998 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6847015-1c4e-4c80-a134-df0a6fc05519-maas-api-tls\") pod \"c6847015-1c4e-4c80-a134-df0a6fc05519\" (UID: \"c6847015-1c4e-4c80-a134-df0a6fc05519\") " Jul 22 19:26:29.360226 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.360199 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6847015-1c4e-4c80-a134-df0a6fc05519-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c6847015-1c4e-4c80-a134-df0a6fc05519" (UID: "c6847015-1c4e-4c80-a134-df0a6fc05519"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:26:29.360324 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.360309 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6847015-1c4e-4c80-a134-df0a6fc05519-openshift-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:26:29.362151 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.362122 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6847015-1c4e-4c80-a134-df0a6fc05519-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c6847015-1c4e-4c80-a134-df0a6fc05519" (UID: "c6847015-1c4e-4c80-a134-df0a6fc05519"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:26:29.362276 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.362214 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6847015-1c4e-4c80-a134-df0a6fc05519-kube-api-access-mqqmr" (OuterVolumeSpecName: "kube-api-access-mqqmr") pod "c6847015-1c4e-4c80-a134-df0a6fc05519" (UID: "c6847015-1c4e-4c80-a134-df0a6fc05519"). InnerVolumeSpecName "kube-api-access-mqqmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:26:29.461293 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.461255 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqqmr\" (UniqueName: \"kubernetes.io/projected/c6847015-1c4e-4c80-a134-df0a6fc05519-kube-api-access-mqqmr\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:26:29.461293 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.461286 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6847015-1c4e-4c80-a134-df0a6fc05519-maas-api-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:26:29.983393 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.983300 2583 generic.go:356] "Generic (PLEG): container finished" podID="c6847015-1c4e-4c80-a134-df0a6fc05519" containerID="4483b1288a8c0017363b8ee0461cc523f99f897b4d7861def7c49a959777a31e" exitCode=0 Jul 22 19:26:29.983393 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.983349 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" Jul 22 19:26:29.983593 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.983386 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" event={"ID":"c6847015-1c4e-4c80-a134-df0a6fc05519","Type":"ContainerDied","Data":"4483b1288a8c0017363b8ee0461cc523f99f897b4d7861def7c49a959777a31e"} Jul 22 19:26:29.983593 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.983440 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh" event={"ID":"c6847015-1c4e-4c80-a134-df0a6fc05519","Type":"ContainerDied","Data":"fdf8c26857d059281d7a34fb9eb9ac6b7e59460243e53b40d6d5b3575700d572"} Jul 22 19:26:29.983593 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.983462 2583 scope.go:117] "RemoveContainer" containerID="4483b1288a8c0017363b8ee0461cc523f99f897b4d7861def7c49a959777a31e" Jul 22 19:26:29.993014 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.992992 2583 scope.go:117] "RemoveContainer" containerID="4483b1288a8c0017363b8ee0461cc523f99f897b4d7861def7c49a959777a31e" Jul 22 19:26:29.993307 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:26:29.993285 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4483b1288a8c0017363b8ee0461cc523f99f897b4d7861def7c49a959777a31e\": container with ID starting with 4483b1288a8c0017363b8ee0461cc523f99f897b4d7861def7c49a959777a31e not found: ID does not exist" containerID="4483b1288a8c0017363b8ee0461cc523f99f897b4d7861def7c49a959777a31e" Jul 22 19:26:29.993388 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:29.993318 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4483b1288a8c0017363b8ee0461cc523f99f897b4d7861def7c49a959777a31e"} err="failed to get container status \"4483b1288a8c0017363b8ee0461cc523f99f897b4d7861def7c49a959777a31e\": rpc error: code = NotFound desc = could not find container \"4483b1288a8c0017363b8ee0461cc523f99f897b4d7861def7c49a959777a31e\": container with ID starting with 4483b1288a8c0017363b8ee0461cc523f99f897b4d7861def7c49a959777a31e not found: ID does not exist" Jul 22 19:26:30.002380 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:30.002349 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh"] Jul 22 19:26:30.006144 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:30.006119 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eaab3c34-694fd7c664-2fwlh"] Jul 22 19:26:31.551125 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:31.551092 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6847015-1c4e-4c80-a134-df0a6fc05519" path="/var/lib/kubelet/pods/c6847015-1c4e-4c80-a134-df0a6fc05519/volumes" Jul 22 19:26:33.703005 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:33.702966 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx"] Jul 22 19:26:33.705656 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:33.705620 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaab3c34-xf2tx"] Jul 22 19:26:35.551107 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:35.551069 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee9e49db-726c-45a1-a068-f674e2137d87" path="/var/lib/kubelet/pods/ee9e49db-726c-45a1-a068-f674e2137d87/volumes" Jul 22 19:26:37.673575 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:37.673541 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:26:37.685077 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:26:37.685052 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:28:14.965848 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:14.965763 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb"] Jul 22 19:28:14.966528 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:14.966487 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee9e49db-726c-45a1-a068-f674e2137d87" containerName="revoke-keys" Jul 22 19:28:14.966601 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:14.966532 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee9e49db-726c-45a1-a068-f674e2137d87" containerName="revoke-keys" Jul 22 19:28:14.966601 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:14.966586 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6847015-1c4e-4c80-a134-df0a6fc05519" containerName="maas-api" Jul 22 19:28:14.966601 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:14.966595 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6847015-1c4e-4c80-a134-df0a6fc05519" containerName="maas-api" Jul 22 19:28:14.966732 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:14.966718 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee9e49db-726c-45a1-a068-f674e2137d87" containerName="revoke-keys" Jul 22 19:28:14.966789 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:14.966737 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6847015-1c4e-4c80-a134-df0a6fc05519" containerName="maas-api" Jul 22 19:28:14.970444 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:14.970420 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" Jul 22 19:28:14.972828 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:14.972808 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c31fc03e-serving-cert\"" Jul 22 19:28:14.972999 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:14.972978 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t4d8t\"" Jul 22 19:28:14.977718 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:14.977693 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb"] Jul 22 19:28:15.082330 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:15.082294 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw58z\" (UniqueName: \"kubernetes.io/projected/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-kube-api-access-nw58z\") pod \"maas-api-e2e-mt-c31fc03e-7655b6788-5njwb\" (UID: \"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" Jul 22 19:28:15.082330 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:15.082335 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-openshift-service-ca\") pod \"maas-api-e2e-mt-c31fc03e-7655b6788-5njwb\" (UID: \"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" Jul 22 19:28:15.082636 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:15.082509 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-maas-api-tls\") pod \"maas-api-e2e-mt-c31fc03e-7655b6788-5njwb\" (UID: \"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" Jul 22 19:28:15.183565 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:15.183528 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw58z\" (UniqueName: \"kubernetes.io/projected/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-kube-api-access-nw58z\") pod \"maas-api-e2e-mt-c31fc03e-7655b6788-5njwb\" (UID: \"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" Jul 22 19:28:15.183565 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:15.183579 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-openshift-service-ca\") pod \"maas-api-e2e-mt-c31fc03e-7655b6788-5njwb\" (UID: \"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" Jul 22 19:28:15.183850 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:15.183671 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-maas-api-tls\") pod \"maas-api-e2e-mt-c31fc03e-7655b6788-5njwb\" (UID: \"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" Jul 22 19:28:15.183850 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:28:15.183765 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-serving-cert: secret "maas-api-e2e-mt-c31fc03e-serving-cert" not found Jul 22 19:28:15.183850 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:28:15.183829 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-maas-api-tls podName:046dbe07-4da4-4ddb-8e16-ff5cbfabfde2 nodeName:}" failed. No retries permitted until 2026-07-22 19:28:15.683812691 +0000 UTC m=+1898.595976737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-maas-api-tls") pod "maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" (UID: "046dbe07-4da4-4ddb-8e16-ff5cbfabfde2") : secret "maas-api-e2e-mt-c31fc03e-serving-cert" not found Jul 22 19:28:15.184288 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:15.184267 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-openshift-service-ca\") pod \"maas-api-e2e-mt-c31fc03e-7655b6788-5njwb\" (UID: \"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" Jul 22 19:28:15.208186 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:15.208159 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw58z\" (UniqueName: \"kubernetes.io/projected/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-kube-api-access-nw58z\") pod \"maas-api-e2e-mt-c31fc03e-7655b6788-5njwb\" (UID: \"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" Jul 22 19:28:15.689956 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:15.689912 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-maas-api-tls\") pod \"maas-api-e2e-mt-c31fc03e-7655b6788-5njwb\" (UID: \"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" Jul 22 19:28:15.692449 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:15.692423 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-maas-api-tls\") pod \"maas-api-e2e-mt-c31fc03e-7655b6788-5njwb\" (UID: \"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" Jul 22 19:28:15.884886 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:15.884841 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" Jul 22 19:28:16.038874 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:16.038840 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb"] Jul 22 19:28:16.039724 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:28:16.039681 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046dbe07_4da4_4ddb_8e16_ff5cbfabfde2.slice/crio-44a92ce10abd17521019f50f40ea69ee77c4829654fe7f81c703b5cc87f6c4cc WatchSource:0}: Error finding container 44a92ce10abd17521019f50f40ea69ee77c4829654fe7f81c703b5cc87f6c4cc: Status 404 returned error can't find the container with id 44a92ce10abd17521019f50f40ea69ee77c4829654fe7f81c703b5cc87f6c4cc Jul 22 19:28:16.620185 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:16.620155 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" event={"ID":"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2","Type":"ContainerStarted","Data":"44a92ce10abd17521019f50f40ea69ee77c4829654fe7f81c703b5cc87f6c4cc"} Jul 22 19:28:17.628095 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:17.628054 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" event={"ID":"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2","Type":"ContainerStarted","Data":"09b4aa778b21f4dda57949d36d88ad0d377b364afaef171b144b63183e664544"} Jul 22 19:28:17.628542 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:17.628191 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" Jul 22 19:28:17.646357 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:17.646304 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" podStartSLOduration=3.07262375 podStartE2EDuration="3.646288831s" totalImagesPullingTime="573.665081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:28:17.643661705 +0000 UTC m=+1900.555825773" watchObservedRunningTime="2026-07-22 19:28:17.646288831 +0000 UTC m=+1900.558452930" Jul 22 19:28:23.640213 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:23.640184 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" Jul 22 19:28:26.753905 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:26.753864 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c"] Jul 22 19:28:26.757816 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:26.757798 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c" Jul 22 19:28:26.760322 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:26.760301 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jhv4j\"" Jul 22 19:28:26.765926 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:26.765674 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c"] Jul 22 19:28:26.791058 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:26.791029 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c\" (UID: \"fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c" Jul 22 19:28:26.892256 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:26.892214 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c\" (UID: \"fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c" Jul 22 19:28:26.892932 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:26.892908 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c\" (UID: \"fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c" Jul 22 19:28:27.070429 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:27.070398 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c" Jul 22 19:28:27.222775 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:27.220892 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c"] Jul 22 19:28:27.223885 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:28:27.223849 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa18bf5e_cb0e_4fea_ac06_3b5e06eeac3f.slice/crio-d0fc581d8dc8d8cbfefc9a21394cfae9a21c170b5939217464ac1042b7c1e047 WatchSource:0}: Error finding container d0fc581d8dc8d8cbfefc9a21394cfae9a21c170b5939217464ac1042b7c1e047: Status 404 returned error can't find the container with id d0fc581d8dc8d8cbfefc9a21394cfae9a21c170b5939217464ac1042b7c1e047 Jul 22 19:28:27.689710 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:27.689626 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c" event={"ID":"fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f","Type":"ContainerStarted","Data":"efaea5733f66cd2fe6a148f406ed315433921109997d5f119801be54568df469"} Jul 22 19:28:27.689710 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:27.689680 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c" event={"ID":"fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f","Type":"ContainerStarted","Data":"d0fc581d8dc8d8cbfefc9a21394cfae9a21c170b5939217464ac1042b7c1e047"} Jul 22 19:28:27.706234 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:27.706178 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c" podStartSLOduration=1.706162143 podStartE2EDuration="1.706162143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:28:27.703323779 +0000 UTC m=+1910.615487848" watchObservedRunningTime="2026-07-22 19:28:27.706162143 +0000 UTC m=+1910.618326210" Jul 22 19:28:28.697459 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:28.697422 2583 generic.go:356] "Generic (PLEG): container finished" podID="fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f" containerID="efaea5733f66cd2fe6a148f406ed315433921109997d5f119801be54568df469" exitCode=0 Jul 22 19:28:28.697907 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:28.697531 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c" event={"ID":"fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f","Type":"ContainerDied","Data":"efaea5733f66cd2fe6a148f406ed315433921109997d5f119801be54568df469"} Jul 22 19:28:29.838769 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:29.838738 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c" Jul 22 19:28:29.921587 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:29.921550 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f-maas-api-service-ca\") pod \"fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f\" (UID: \"fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f\") " Jul 22 19:28:29.921895 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:29.921872 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f" (UID: "fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:28:30.022728 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:30.022682 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:28:30.711566 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:30.711533 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c" Jul 22 19:28:30.711743 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:30.711532 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c" event={"ID":"fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f","Type":"ContainerDied","Data":"d0fc581d8dc8d8cbfefc9a21394cfae9a21c170b5939217464ac1042b7c1e047"} Jul 22 19:28:30.711743 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:30.711658 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0fc581d8dc8d8cbfefc9a21394cfae9a21c170b5939217464ac1042b7c1e047" Jul 22 19:28:37.134038 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.134003 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb"] Jul 22 19:28:37.134549 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.134251 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" podUID="046dbe07-4da4-4ddb-8e16-ff5cbfabfde2" containerName="maas-api" containerID="cri-o://09b4aa778b21f4dda57949d36d88ad0d377b364afaef171b144b63183e664544" gracePeriod=30 Jul 22 19:28:37.384221 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.384150 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" Jul 22 19:28:37.493027 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.492997 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw58z\" (UniqueName: \"kubernetes.io/projected/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-kube-api-access-nw58z\") pod \"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2\" (UID: \"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2\") " Jul 22 19:28:37.493228 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.493087 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-openshift-service-ca\") pod \"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2\" (UID: \"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2\") " Jul 22 19:28:37.493228 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.493136 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-maas-api-tls\") pod \"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2\" (UID: \"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2\") " Jul 22 19:28:37.493411 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.493384 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "046dbe07-4da4-4ddb-8e16-ff5cbfabfde2" (UID: "046dbe07-4da4-4ddb-8e16-ff5cbfabfde2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:28:37.495217 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.495192 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-kube-api-access-nw58z" (OuterVolumeSpecName: "kube-api-access-nw58z") pod "046dbe07-4da4-4ddb-8e16-ff5cbfabfde2" (UID: "046dbe07-4da4-4ddb-8e16-ff5cbfabfde2"). InnerVolumeSpecName "kube-api-access-nw58z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:28:37.495333 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.495311 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "046dbe07-4da4-4ddb-8e16-ff5cbfabfde2" (UID: "046dbe07-4da4-4ddb-8e16-ff5cbfabfde2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:28:37.594688 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.594650 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-openshift-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:28:37.594688 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.594684 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-maas-api-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:28:37.594890 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.594698 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw58z\" (UniqueName: \"kubernetes.io/projected/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2-kube-api-access-nw58z\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:28:37.755914 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.755823 2583 generic.go:356] "Generic (PLEG): container finished" podID="046dbe07-4da4-4ddb-8e16-ff5cbfabfde2" containerID="09b4aa778b21f4dda57949d36d88ad0d377b364afaef171b144b63183e664544" exitCode=0 Jul 22 19:28:37.755914 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.755880 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" Jul 22 19:28:37.755914 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.755900 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" event={"ID":"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2","Type":"ContainerDied","Data":"09b4aa778b21f4dda57949d36d88ad0d377b364afaef171b144b63183e664544"} Jul 22 19:28:37.756141 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.755954 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb" event={"ID":"046dbe07-4da4-4ddb-8e16-ff5cbfabfde2","Type":"ContainerDied","Data":"44a92ce10abd17521019f50f40ea69ee77c4829654fe7f81c703b5cc87f6c4cc"} Jul 22 19:28:37.756141 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.755976 2583 scope.go:117] "RemoveContainer" containerID="09b4aa778b21f4dda57949d36d88ad0d377b364afaef171b144b63183e664544" Jul 22 19:28:37.765882 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.765860 2583 scope.go:117] "RemoveContainer" containerID="09b4aa778b21f4dda57949d36d88ad0d377b364afaef171b144b63183e664544" Jul 22 19:28:37.766157 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:28:37.766141 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09b4aa778b21f4dda57949d36d88ad0d377b364afaef171b144b63183e664544\": container with ID starting with 09b4aa778b21f4dda57949d36d88ad0d377b364afaef171b144b63183e664544 not found: ID does not exist" containerID="09b4aa778b21f4dda57949d36d88ad0d377b364afaef171b144b63183e664544" Jul 22 19:28:37.766224 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.766168 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b4aa778b21f4dda57949d36d88ad0d377b364afaef171b144b63183e664544"} err="failed to get container status \"09b4aa778b21f4dda57949d36d88ad0d377b364afaef171b144b63183e664544\": rpc error: code = NotFound desc = could not find container \"09b4aa778b21f4dda57949d36d88ad0d377b364afaef171b144b63183e664544\": container with ID starting with 09b4aa778b21f4dda57949d36d88ad0d377b364afaef171b144b63183e664544 not found: ID does not exist" Jul 22 19:28:37.773651 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.773625 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb"] Jul 22 19:28:37.777609 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:37.777569 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c31fc03e-7655b6788-5njwb"] Jul 22 19:28:39.551052 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:39.551014 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="046dbe07-4da4-4ddb-8e16-ff5cbfabfde2" path="/var/lib/kubelet/pods/046dbe07-4da4-4ddb-8e16-ff5cbfabfde2/volumes" Jul 22 19:28:41.828349 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:41.828307 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c"] Jul 22 19:28:41.831022 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:41.830997 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c31fc03e-xkn6c"] Jul 22 19:28:43.550987 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:28:43.550956 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f" path="/var/lib/kubelet/pods/fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f/volumes" Jul 22 19:29:11.254806 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.254771 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn"] Jul 22 19:29:11.257296 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.255364 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046dbe07-4da4-4ddb-8e16-ff5cbfabfde2" containerName="maas-api" Jul 22 19:29:11.257296 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.255383 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="046dbe07-4da4-4ddb-8e16-ff5cbfabfde2" containerName="maas-api" Jul 22 19:29:11.257296 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.255394 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f" containerName="revoke-keys" Jul 22 19:29:11.257296 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.255399 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f" containerName="revoke-keys" Jul 22 19:29:11.257296 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.255480 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa18bf5e-cb0e-4fea-ac06-3b5e06eeac3f" containerName="revoke-keys" Jul 22 19:29:11.257296 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.255505 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="046dbe07-4da4-4ddb-8e16-ff5cbfabfde2" containerName="maas-api" Jul 22 19:29:11.258545 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.258528 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" Jul 22 19:29:11.261084 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.261064 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7a0d319a-serving-cert\"" Jul 22 19:29:11.261084 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.261078 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t4d8t\"" Jul 22 19:29:11.269163 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.269139 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn"] Jul 22 19:29:11.325960 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.325924 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c4376f8-538a-4fab-bffa-01c174dffb89-maas-api-tls\") pod \"maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn\" (UID: \"1c4376f8-538a-4fab-bffa-01c174dffb89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" Jul 22 19:29:11.325960 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.325957 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qwzh\" (UniqueName: \"kubernetes.io/projected/1c4376f8-538a-4fab-bffa-01c174dffb89-kube-api-access-4qwzh\") pod \"maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn\" (UID: \"1c4376f8-538a-4fab-bffa-01c174dffb89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" Jul 22 19:29:11.326168 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.326058 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4376f8-538a-4fab-bffa-01c174dffb89-openshift-service-ca\") pod \"maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn\" (UID: \"1c4376f8-538a-4fab-bffa-01c174dffb89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" Jul 22 19:29:11.427553 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.427515 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c4376f8-538a-4fab-bffa-01c174dffb89-maas-api-tls\") pod \"maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn\" (UID: \"1c4376f8-538a-4fab-bffa-01c174dffb89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" Jul 22 19:29:11.427553 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.427554 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qwzh\" (UniqueName: \"kubernetes.io/projected/1c4376f8-538a-4fab-bffa-01c174dffb89-kube-api-access-4qwzh\") pod \"maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn\" (UID: \"1c4376f8-538a-4fab-bffa-01c174dffb89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" Jul 22 19:29:11.427766 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.427608 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4376f8-538a-4fab-bffa-01c174dffb89-openshift-service-ca\") pod \"maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn\" (UID: \"1c4376f8-538a-4fab-bffa-01c174dffb89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" Jul 22 19:29:11.428252 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.428232 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4376f8-538a-4fab-bffa-01c174dffb89-openshift-service-ca\") pod \"maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn\" (UID: \"1c4376f8-538a-4fab-bffa-01c174dffb89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" Jul 22 19:29:11.430169 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.430142 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c4376f8-538a-4fab-bffa-01c174dffb89-maas-api-tls\") pod \"maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn\" (UID: \"1c4376f8-538a-4fab-bffa-01c174dffb89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" Jul 22 19:29:11.436220 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.436193 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qwzh\" (UniqueName: \"kubernetes.io/projected/1c4376f8-538a-4fab-bffa-01c174dffb89-kube-api-access-4qwzh\") pod \"maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn\" (UID: \"1c4376f8-538a-4fab-bffa-01c174dffb89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" Jul 22 19:29:11.570798 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.570765 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" Jul 22 19:29:11.718659 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.718630 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn"] Jul 22 19:29:11.719890 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:29:11.719866 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4376f8_538a_4fab_bffa_01c174dffb89.slice/crio-1d5f6e146b4017c3214cb33608c9f8584a60d839e6fb8da013117a6a036dea33 WatchSource:0}: Error finding container 1d5f6e146b4017c3214cb33608c9f8584a60d839e6fb8da013117a6a036dea33: Status 404 returned error can't find the container with id 1d5f6e146b4017c3214cb33608c9f8584a60d839e6fb8da013117a6a036dea33 Jul 22 19:29:11.721604 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.721315 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 19:29:11.967751 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:11.967664 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" event={"ID":"1c4376f8-538a-4fab-bffa-01c174dffb89","Type":"ContainerStarted","Data":"1d5f6e146b4017c3214cb33608c9f8584a60d839e6fb8da013117a6a036dea33"} Jul 22 19:29:12.975987 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:12.975953 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" event={"ID":"1c4376f8-538a-4fab-bffa-01c174dffb89","Type":"ContainerStarted","Data":"95b74a1e9fc724d3e8cc6b8c3c798670da612f36e09482fa6539c3f78415b9fb"} Jul 22 19:29:12.976479 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:12.976014 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" Jul 22 19:29:12.992269 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:12.992211 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" podStartSLOduration=1.634005294 podStartE2EDuration="1.992195338s" totalImagesPullingTime="358.190044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:29:12.991345012 +0000 UTC m=+1955.903509080" watchObservedRunningTime="2026-07-22 19:29:12.992195338 +0000 UTC m=+1955.904359405" Jul 22 19:29:18.988218 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:18.988188 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" Jul 22 19:29:53.949025 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:53.948986 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj"] Jul 22 19:29:53.952836 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:53.952818 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj" Jul 22 19:29:53.956339 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:53.956317 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jhv4j\"" Jul 22 19:29:53.966992 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:53.966959 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj"] Jul 22 19:29:54.049485 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:54.049447 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b4ecd2-1158-41af-81f9-2784e6a07cc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj\" (UID: \"b8b4ecd2-1158-41af-81f9-2784e6a07cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj" Jul 22 19:29:54.150419 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:54.150381 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b4ecd2-1158-41af-81f9-2784e6a07cc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj\" (UID: \"b8b4ecd2-1158-41af-81f9-2784e6a07cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj" Jul 22 19:29:54.151064 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:54.151031 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b4ecd2-1158-41af-81f9-2784e6a07cc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj\" (UID: \"b8b4ecd2-1158-41af-81f9-2784e6a07cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj" Jul 22 19:29:54.264093 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:54.263998 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj" Jul 22 19:29:54.412423 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:54.412396 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj"] Jul 22 19:29:54.413568 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:29:54.413536 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b4ecd2_1158_41af_81f9_2784e6a07cc7.slice/crio-69c5fa3c63e53347bc395a6d2a8df954d10b42c575884d50a77a1e4ca02fe49b WatchSource:0}: Error finding container 69c5fa3c63e53347bc395a6d2a8df954d10b42c575884d50a77a1e4ca02fe49b: Status 404 returned error can't find the container with id 69c5fa3c63e53347bc395a6d2a8df954d10b42c575884d50a77a1e4ca02fe49b Jul 22 19:29:55.247909 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:55.247865 2583 generic.go:356] "Generic (PLEG): container finished" podID="b8b4ecd2-1158-41af-81f9-2784e6a07cc7" containerID="5b705011ade99d86150fb1a1ef4102ae0a0a8f3318fc851b113c3a3a0c35dfe4" exitCode=0 Jul 22 19:29:55.248290 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:55.247974 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj" event={"ID":"b8b4ecd2-1158-41af-81f9-2784e6a07cc7","Type":"ContainerDied","Data":"5b705011ade99d86150fb1a1ef4102ae0a0a8f3318fc851b113c3a3a0c35dfe4"} Jul 22 19:29:55.248290 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:55.248039 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj" event={"ID":"b8b4ecd2-1158-41af-81f9-2784e6a07cc7","Type":"ContainerStarted","Data":"69c5fa3c63e53347bc395a6d2a8df954d10b42c575884d50a77a1e4ca02fe49b"} Jul 22 19:29:56.415657 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:56.415633 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj" Jul 22 19:29:56.474910 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:56.474878 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b4ecd2-1158-41af-81f9-2784e6a07cc7-maas-api-service-ca\") pod \"b8b4ecd2-1158-41af-81f9-2784e6a07cc7\" (UID: \"b8b4ecd2-1158-41af-81f9-2784e6a07cc7\") " Jul 22 19:29:56.475232 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:56.475210 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b4ecd2-1158-41af-81f9-2784e6a07cc7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b8b4ecd2-1158-41af-81f9-2784e6a07cc7" (UID: "b8b4ecd2-1158-41af-81f9-2784e6a07cc7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:29:56.576474 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:56.576439 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b4ecd2-1158-41af-81f9-2784e6a07cc7-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:29:57.261557 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:57.261530 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj" Jul 22 19:29:57.261771 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:57.261593 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj" event={"ID":"b8b4ecd2-1158-41af-81f9-2784e6a07cc7","Type":"ContainerDied","Data":"69c5fa3c63e53347bc395a6d2a8df954d10b42c575884d50a77a1e4ca02fe49b"} Jul 22 19:29:57.261771 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:29:57.261639 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69c5fa3c63e53347bc395a6d2a8df954d10b42c575884d50a77a1e4ca02fe49b" Jul 22 19:30:00.130787 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:00.130752 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745810-75xxn"] Jul 22 19:30:00.131236 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:00.131221 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b4ecd2-1158-41af-81f9-2784e6a07cc7" containerName="revoke-keys" Jul 22 19:30:00.131281 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:00.131238 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b4ecd2-1158-41af-81f9-2784e6a07cc7" containerName="revoke-keys" Jul 22 19:30:00.131345 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:00.131336 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b4ecd2-1158-41af-81f9-2784e6a07cc7" containerName="revoke-keys" Jul 22 19:30:00.134541 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:00.134520 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a0d319a-29745810-g25hd"] Jul 22 19:30:00.134651 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:00.134615 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745810-75xxn" Jul 22 19:30:00.137201 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:00.137104 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jhv4j\"" Jul 22 19:30:00.138048 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:00.138026 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a0d319a-29745810-g25hd" Jul 22 19:30:00.145851 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:00.142384 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745810-75xxn"] Jul 22 19:30:00.145851 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:00.145547 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a0d319a-29745810-g25hd"] Jul 22 19:30:00.149132 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:00.149110 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745810-75xxn" Jul 22 19:30:00.157289 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:00.157257 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a0d319a-29745810-g25hd" Jul 22 19:30:00.307943 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:30:00.307897 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69fce0b_0962_4daf_aa4e_189d9361903d.slice/crio-69dd03c6ad5be270ffd3feebc8f1c53e98615f9d6d78a9998d0e7f9d59d066d6 WatchSource:0}: Error finding container 69dd03c6ad5be270ffd3feebc8f1c53e98615f9d6d78a9998d0e7f9d59d066d6: Status 404 returned error can't find the container with id 69dd03c6ad5be270ffd3feebc8f1c53e98615f9d6d78a9998d0e7f9d59d066d6 Jul 22 19:30:00.308437 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:00.308406 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745810-75xxn"] Jul 22 19:30:00.341256 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:00.341223 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a0d319a-29745810-g25hd"] Jul 22 19:30:00.343252 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:30:00.343229 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246d64d7_24eb_45f2_b539_e3a21ee61251.slice/crio-a59eacebc49ee0c58364e7399e43ecb245f05e5b0f6316c2bbca43858e6afaaa WatchSource:0}: Error finding container a59eacebc49ee0c58364e7399e43ecb245f05e5b0f6316c2bbca43858e6afaaa: Status 404 returned error can't find the container with id a59eacebc49ee0c58364e7399e43ecb245f05e5b0f6316c2bbca43858e6afaaa Jul 22 19:30:01.292223 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:01.292193 2583 generic.go:356] "Generic (PLEG): container finished" podID="b69fce0b-0962-4daf-aa4e-189d9361903d" containerID="94666654f4ed2b9318f0b3ea8b9cb966a148b3b4f197946ca8aa1bb51bd74d07" exitCode=0 Jul 22 19:30:01.292613 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:01.292281 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745810-75xxn" event={"ID":"b69fce0b-0962-4daf-aa4e-189d9361903d","Type":"ContainerDied","Data":"94666654f4ed2b9318f0b3ea8b9cb966a148b3b4f197946ca8aa1bb51bd74d07"} Jul 22 19:30:01.292613 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:01.292343 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745810-75xxn" event={"ID":"b69fce0b-0962-4daf-aa4e-189d9361903d","Type":"ContainerStarted","Data":"69dd03c6ad5be270ffd3feebc8f1c53e98615f9d6d78a9998d0e7f9d59d066d6"} Jul 22 19:30:01.293760 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:01.293732 2583 generic.go:356] "Generic (PLEG): container finished" podID="246d64d7-24eb-45f2-b539-e3a21ee61251" containerID="6413956e86420277648bdf0f8e3460505032f67c38cfb49983312f10b9c87f58" exitCode=0 Jul 22 19:30:01.293872 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:01.293820 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a0d319a-29745810-g25hd" event={"ID":"246d64d7-24eb-45f2-b539-e3a21ee61251","Type":"ContainerDied","Data":"6413956e86420277648bdf0f8e3460505032f67c38cfb49983312f10b9c87f58"} Jul 22 19:30:01.293872 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:01.293853 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a0d319a-29745810-g25hd" event={"ID":"246d64d7-24eb-45f2-b539-e3a21ee61251","Type":"ContainerStarted","Data":"a59eacebc49ee0c58364e7399e43ecb245f05e5b0f6316c2bbca43858e6afaaa"} Jul 22 19:30:02.445516 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:02.445470 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745810-75xxn" Jul 22 19:30:02.468883 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:02.468859 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a0d319a-29745810-g25hd" Jul 22 19:30:03.307474 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:03.307442 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745810-75xxn" Jul 22 19:30:03.307474 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:03.307451 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745810-75xxn" event={"ID":"b69fce0b-0962-4daf-aa4e-189d9361903d","Type":"ContainerDied","Data":"69dd03c6ad5be270ffd3feebc8f1c53e98615f9d6d78a9998d0e7f9d59d066d6"} Jul 22 19:30:03.307720 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:03.307524 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69dd03c6ad5be270ffd3feebc8f1c53e98615f9d6d78a9998d0e7f9d59d066d6" Jul 22 19:30:03.308866 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:03.308846 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a0d319a-29745810-g25hd" Jul 22 19:30:03.308977 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:03.308875 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a0d319a-29745810-g25hd" event={"ID":"246d64d7-24eb-45f2-b539-e3a21ee61251","Type":"ContainerDied","Data":"a59eacebc49ee0c58364e7399e43ecb245f05e5b0f6316c2bbca43858e6afaaa"} Jul 22 19:30:03.308977 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:03.308912 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a59eacebc49ee0c58364e7399e43ecb245f05e5b0f6316c2bbca43858e6afaaa" Jul 22 19:30:09.359349 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:09.359314 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn"] Jul 22 19:30:09.359815 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:09.359588 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" podUID="1c4376f8-538a-4fab-bffa-01c174dffb89" containerName="maas-api" containerID="cri-o://95b74a1e9fc724d3e8cc6b8c3c798670da612f36e09482fa6539c3f78415b9fb" gracePeriod=30 Jul 22 19:30:09.386388 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:09.386337 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a0d319a-29745810-g25hd"] Jul 22 19:30:09.388943 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:09.388918 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a0d319a-29745810-g25hd"] Jul 22 19:30:09.552853 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:09.552819 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="246d64d7-24eb-45f2-b539-e3a21ee61251" path="/var/lib/kubelet/pods/246d64d7-24eb-45f2-b539-e3a21ee61251/volumes" Jul 22 19:30:09.620604 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:09.615994 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" Jul 22 19:30:09.701974 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:09.701937 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c4376f8-538a-4fab-bffa-01c174dffb89-maas-api-tls\") pod \"1c4376f8-538a-4fab-bffa-01c174dffb89\" (UID: \"1c4376f8-538a-4fab-bffa-01c174dffb89\") " Jul 22 19:30:09.702144 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:09.702034 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qwzh\" (UniqueName: \"kubernetes.io/projected/1c4376f8-538a-4fab-bffa-01c174dffb89-kube-api-access-4qwzh\") pod \"1c4376f8-538a-4fab-bffa-01c174dffb89\" (UID: \"1c4376f8-538a-4fab-bffa-01c174dffb89\") " Jul 22 19:30:09.702144 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:09.702074 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4376f8-538a-4fab-bffa-01c174dffb89-openshift-service-ca\") pod \"1c4376f8-538a-4fab-bffa-01c174dffb89\" (UID: \"1c4376f8-538a-4fab-bffa-01c174dffb89\") " Jul 22 19:30:09.702534 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:09.702477 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4376f8-538a-4fab-bffa-01c174dffb89-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1c4376f8-538a-4fab-bffa-01c174dffb89" (UID: "1c4376f8-538a-4fab-bffa-01c174dffb89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:30:09.704273 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:09.704247 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4376f8-538a-4fab-bffa-01c174dffb89-kube-api-access-4qwzh" (OuterVolumeSpecName: "kube-api-access-4qwzh") pod "1c4376f8-538a-4fab-bffa-01c174dffb89" (UID: "1c4376f8-538a-4fab-bffa-01c174dffb89"). InnerVolumeSpecName "kube-api-access-4qwzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:30:09.704406 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:09.704385 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4376f8-538a-4fab-bffa-01c174dffb89-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1c4376f8-538a-4fab-bffa-01c174dffb89" (UID: "1c4376f8-538a-4fab-bffa-01c174dffb89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:30:09.802998 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:09.802965 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qwzh\" (UniqueName: \"kubernetes.io/projected/1c4376f8-538a-4fab-bffa-01c174dffb89-kube-api-access-4qwzh\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:30:09.802998 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:09.802997 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4376f8-538a-4fab-bffa-01c174dffb89-openshift-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:30:09.802998 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:09.803008 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c4376f8-538a-4fab-bffa-01c174dffb89-maas-api-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:30:10.355776 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:10.355738 2583 generic.go:356] "Generic (PLEG): container finished" podID="1c4376f8-538a-4fab-bffa-01c174dffb89" containerID="95b74a1e9fc724d3e8cc6b8c3c798670da612f36e09482fa6539c3f78415b9fb" exitCode=0 Jul 22 19:30:10.355932 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:10.355789 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" Jul 22 19:30:10.355932 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:10.355830 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" event={"ID":"1c4376f8-538a-4fab-bffa-01c174dffb89","Type":"ContainerDied","Data":"95b74a1e9fc724d3e8cc6b8c3c798670da612f36e09482fa6539c3f78415b9fb"} Jul 22 19:30:10.355932 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:10.355910 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn" event={"ID":"1c4376f8-538a-4fab-bffa-01c174dffb89","Type":"ContainerDied","Data":"1d5f6e146b4017c3214cb33608c9f8584a60d839e6fb8da013117a6a036dea33"} Jul 22 19:30:10.355932 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:10.355932 2583 scope.go:117] "RemoveContainer" containerID="95b74a1e9fc724d3e8cc6b8c3c798670da612f36e09482fa6539c3f78415b9fb" Jul 22 19:30:10.367114 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:10.366914 2583 scope.go:117] "RemoveContainer" containerID="95b74a1e9fc724d3e8cc6b8c3c798670da612f36e09482fa6539c3f78415b9fb" Jul 22 19:30:10.367484 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:30:10.367228 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b74a1e9fc724d3e8cc6b8c3c798670da612f36e09482fa6539c3f78415b9fb\": container with ID starting with 95b74a1e9fc724d3e8cc6b8c3c798670da612f36e09482fa6539c3f78415b9fb not found: ID does not exist" containerID="95b74a1e9fc724d3e8cc6b8c3c798670da612f36e09482fa6539c3f78415b9fb" Jul 22 19:30:10.367484 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:10.367266 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b74a1e9fc724d3e8cc6b8c3c798670da612f36e09482fa6539c3f78415b9fb"} err="failed to get container status \"95b74a1e9fc724d3e8cc6b8c3c798670da612f36e09482fa6539c3f78415b9fb\": rpc error: code = NotFound desc = could not find container \"95b74a1e9fc724d3e8cc6b8c3c798670da612f36e09482fa6539c3f78415b9fb\": container with ID starting with 95b74a1e9fc724d3e8cc6b8c3c798670da612f36e09482fa6539c3f78415b9fb not found: ID does not exist" Jul 22 19:30:10.378659 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:10.378604 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn"] Jul 22 19:30:10.381926 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:10.381902 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a0d319a-6589c96b9b-8sjnn"] Jul 22 19:30:11.551309 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:11.551274 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c4376f8-538a-4fab-bffa-01c174dffb89" path="/var/lib/kubelet/pods/1c4376f8-538a-4fab-bffa-01c174dffb89/volumes" Jul 22 19:30:14.018633 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:14.018592 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj"] Jul 22 19:30:14.021227 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:14.021203 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a0d319a-dkknj"] Jul 22 19:30:15.551147 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:15.551114 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b4ecd2-1158-41af-81f9-2784e6a07cc7" path="/var/lib/kubelet/pods/b8b4ecd2-1158-41af-81f9-2784e6a07cc7/volumes" Jul 22 19:30:37.807693 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:30:37.807658 2583 scope.go:117] "RemoveContainer" containerID="577f8d6e6ea10f457d20555ca2a6cba0a137e1944df98084c58f1a6aaaf690c8" Jul 22 19:31:37.716431 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:31:37.716401 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:31:37.728536 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:31:37.728486 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:31:37.839241 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:31:37.839210 2583 scope.go:117] "RemoveContainer" containerID="8252d56aeefd2e816ad628c8d34712fccf3eeedf0edb7b4483011767d56f407e" Jul 22 19:31:37.848576 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:31:37.848550 2583 scope.go:117] "RemoveContainer" containerID="244e203b37f92f78893714d02718ee5ce72312057123e633cf53c60ccb170402" Jul 22 19:32:01.911973 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.911936 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6"] Jul 22 19:32:01.912440 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.912394 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c4376f8-538a-4fab-bffa-01c174dffb89" containerName="maas-api" Jul 22 19:32:01.912440 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.912406 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c4376f8-538a-4fab-bffa-01c174dffb89" containerName="maas-api" Jul 22 19:32:01.912440 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.912422 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="246d64d7-24eb-45f2-b539-e3a21ee61251" containerName="cleanup" Jul 22 19:32:01.912440 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.912427 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="246d64d7-24eb-45f2-b539-e3a21ee61251" containerName="cleanup" Jul 22 19:32:01.912599 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.912443 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b69fce0b-0962-4daf-aa4e-189d9361903d" containerName="cleanup" Jul 22 19:32:01.912599 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.912449 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b69fce0b-0962-4daf-aa4e-189d9361903d" containerName="cleanup" Jul 22 19:32:01.912599 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.912537 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c4376f8-538a-4fab-bffa-01c174dffb89" containerName="maas-api" Jul 22 19:32:01.912599 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.912546 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="246d64d7-24eb-45f2-b539-e3a21ee61251" containerName="cleanup" Jul 22 19:32:01.912599 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.912553 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b69fce0b-0962-4daf-aa4e-189d9361903d" containerName="cleanup" Jul 22 19:32:01.914965 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.914949 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" Jul 22 19:32:01.917788 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.917761 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-735d50-serving-cert\"" Jul 22 19:32:01.917788 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.917780 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t4d8t\"" Jul 22 19:32:01.926257 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.926233 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6"] Jul 22 19:32:01.983157 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.983122 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-openshift-service-ca\") pod \"maas-api-e2e-inf-a-735d50-f77f6df88-825s6\" (UID: \"ec1e9e4b-f81b-40d8-b85f-19e549ad95de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" Jul 22 19:32:01.983352 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.983170 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-maas-api-tls\") pod \"maas-api-e2e-inf-a-735d50-f77f6df88-825s6\" (UID: \"ec1e9e4b-f81b-40d8-b85f-19e549ad95de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" Jul 22 19:32:01.983352 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:01.983269 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c8dq\" (UniqueName: \"kubernetes.io/projected/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-kube-api-access-9c8dq\") pod \"maas-api-e2e-inf-a-735d50-f77f6df88-825s6\" (UID: \"ec1e9e4b-f81b-40d8-b85f-19e549ad95de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" Jul 22 19:32:02.084862 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:02.084817 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c8dq\" (UniqueName: \"kubernetes.io/projected/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-kube-api-access-9c8dq\") pod \"maas-api-e2e-inf-a-735d50-f77f6df88-825s6\" (UID: \"ec1e9e4b-f81b-40d8-b85f-19e549ad95de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" Jul 22 19:32:02.085070 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:02.084928 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-openshift-service-ca\") pod \"maas-api-e2e-inf-a-735d50-f77f6df88-825s6\" (UID: \"ec1e9e4b-f81b-40d8-b85f-19e549ad95de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" Jul 22 19:32:02.085070 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:02.084974 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-maas-api-tls\") pod \"maas-api-e2e-inf-a-735d50-f77f6df88-825s6\" (UID: \"ec1e9e4b-f81b-40d8-b85f-19e549ad95de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" Jul 22 19:32:02.085641 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:02.085617 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-openshift-service-ca\") pod \"maas-api-e2e-inf-a-735d50-f77f6df88-825s6\" (UID: \"ec1e9e4b-f81b-40d8-b85f-19e549ad95de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" Jul 22 19:32:02.087645 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:02.087614 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-maas-api-tls\") pod \"maas-api-e2e-inf-a-735d50-f77f6df88-825s6\" (UID: \"ec1e9e4b-f81b-40d8-b85f-19e549ad95de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" Jul 22 19:32:02.096641 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:02.096613 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c8dq\" (UniqueName: \"kubernetes.io/projected/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-kube-api-access-9c8dq\") pod \"maas-api-e2e-inf-a-735d50-f77f6df88-825s6\" (UID: \"ec1e9e4b-f81b-40d8-b85f-19e549ad95de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" Jul 22 19:32:02.227287 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:02.227194 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" Jul 22 19:32:02.373989 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:02.373953 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6"] Jul 22 19:32:02.374365 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:32:02.374335 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1e9e4b_f81b_40d8_b85f_19e549ad95de.slice/crio-b7fdc54076df5d3782b7efac6241eaf6d6624ef98ff57f36c10febea33e87c74 WatchSource:0}: Error finding container b7fdc54076df5d3782b7efac6241eaf6d6624ef98ff57f36c10febea33e87c74: Status 404 returned error can't find the container with id b7fdc54076df5d3782b7efac6241eaf6d6624ef98ff57f36c10febea33e87c74 Jul 22 19:32:03.056350 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:03.056313 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" event={"ID":"ec1e9e4b-f81b-40d8-b85f-19e549ad95de","Type":"ContainerStarted","Data":"b7fdc54076df5d3782b7efac6241eaf6d6624ef98ff57f36c10febea33e87c74"} Jul 22 19:32:04.065260 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:04.065227 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" event={"ID":"ec1e9e4b-f81b-40d8-b85f-19e549ad95de","Type":"ContainerStarted","Data":"81066b7652a447e6c998a6e1547e359dba3d3c7f4ae83a00c75d1e8164e76f99"} Jul 22 19:32:04.065676 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:04.065362 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" Jul 22 19:32:04.081060 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:04.081009 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" podStartSLOduration=1.9340712500000001 podStartE2EDuration="3.080993768s" totalImagesPullingTime="1.146922518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:32:04.080002394 +0000 UTC m=+2126.992166463" watchObservedRunningTime="2026-07-22 19:32:04.080993768 +0000 UTC m=+2126.993157904" Jul 22 19:32:09.979112 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:09.979032 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm"] Jul 22 19:32:09.983062 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:09.983042 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" Jul 22 19:32:09.986329 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:09.986305 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-dc2828-serving-cert\"" Jul 22 19:32:09.997074 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:09.997048 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm"] Jul 22 19:32:10.066249 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:10.066216 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68fm6\" (UniqueName: \"kubernetes.io/projected/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-kube-api-access-68fm6\") pod \"maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm\" (UID: \"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" Jul 22 19:32:10.066465 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:10.066257 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm\" (UID: \"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" Jul 22 19:32:10.066465 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:10.066380 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-maas-api-tls\") pod \"maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm\" (UID: \"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" Jul 22 19:32:10.077987 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:10.077961 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" Jul 22 19:32:10.167508 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:10.167463 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm\" (UID: \"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" Jul 22 19:32:10.167898 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:10.167870 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-maas-api-tls\") pod \"maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm\" (UID: \"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" Jul 22 19:32:10.168187 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:10.168025 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68fm6\" (UniqueName: \"kubernetes.io/projected/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-kube-api-access-68fm6\") pod \"maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm\" (UID: \"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" Jul 22 19:32:10.169254 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:10.169228 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm\" (UID: \"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" Jul 22 19:32:10.172070 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:10.172047 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-maas-api-tls\") pod \"maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm\" (UID: \"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" Jul 22 19:32:10.176572 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:10.176550 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68fm6\" (UniqueName: \"kubernetes.io/projected/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-kube-api-access-68fm6\") pod \"maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm\" (UID: \"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" Jul 22 19:32:10.295309 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:10.295266 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" Jul 22 19:32:10.445118 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:10.445088 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm"] Jul 22 19:32:10.446620 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:32:10.446582 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68be4238_55c0_43e9_aa06_a7ef5e3d6ca2.slice/crio-839caad6e594d034990c7abd836eb0699bb4c6a8f2bb2ffc1c5dd36ac35deecc WatchSource:0}: Error finding container 839caad6e594d034990c7abd836eb0699bb4c6a8f2bb2ffc1c5dd36ac35deecc: Status 404 returned error can't find the container with id 839caad6e594d034990c7abd836eb0699bb4c6a8f2bb2ffc1c5dd36ac35deecc Jul 22 19:32:11.120904 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:11.120852 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" event={"ID":"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2","Type":"ContainerStarted","Data":"078baae562924767e651c3ac5d41ae90c11a3a671cbef239e9fa80df7e5e29e4"} Jul 22 19:32:11.121319 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:11.120955 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" Jul 22 19:32:11.121319 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:11.120979 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" event={"ID":"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2","Type":"ContainerStarted","Data":"839caad6e594d034990c7abd836eb0699bb4c6a8f2bb2ffc1c5dd36ac35deecc"} Jul 22 19:32:11.141170 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:11.141115 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" podStartSLOduration=1.625203984 podStartE2EDuration="2.141099429s" totalImagesPullingTime="515.895445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:32:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:32:11.138470947 +0000 UTC m=+2134.050635017" watchObservedRunningTime="2026-07-22 19:32:11.141099429 +0000 UTC m=+2134.053263520" Jul 22 19:32:17.132837 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:17.132810 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" Jul 22 19:32:37.878059 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:32:37.877977 2583 scope.go:117] "RemoveContainer" containerID="3f83e25122fcc073cda6f7d35df5ebad41626690c98821cb456878fc3bfc2f55" Jul 22 19:33:51.105407 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:51.105370 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q"] Jul 22 19:33:51.109053 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:51.109034 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q" Jul 22 19:33:51.111488 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:51.111461 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jhv4j\"" Jul 22 19:33:51.117150 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:51.117120 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q"] Jul 22 19:33:51.156361 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:51.156323 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01a4fa3-4595-4c77-a6e9-82278e0bf264-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q\" (UID: \"b01a4fa3-4595-4c77-a6e9-82278e0bf264\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q" Jul 22 19:33:51.257806 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:51.257765 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01a4fa3-4595-4c77-a6e9-82278e0bf264-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q\" (UID: \"b01a4fa3-4595-4c77-a6e9-82278e0bf264\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q" Jul 22 19:33:51.258390 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:51.258369 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01a4fa3-4595-4c77-a6e9-82278e0bf264-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q\" (UID: \"b01a4fa3-4595-4c77-a6e9-82278e0bf264\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q" Jul 22 19:33:51.424342 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:51.424244 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q" Jul 22 19:33:51.568699 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:51.568672 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q"] Jul 22 19:33:51.569987 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:33:51.569956 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01a4fa3_4595_4c77_a6e9_82278e0bf264.slice/crio-12d3b8949ca55b2e9a347c0a483b12e4b7b21c7794ef16ca85fe941ce45e015e WatchSource:0}: Error finding container 12d3b8949ca55b2e9a347c0a483b12e4b7b21c7794ef16ca85fe941ce45e015e: Status 404 returned error can't find the container with id 12d3b8949ca55b2e9a347c0a483b12e4b7b21c7794ef16ca85fe941ce45e015e Jul 22 19:33:51.767944 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:51.767901 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q" event={"ID":"b01a4fa3-4595-4c77-a6e9-82278e0bf264","Type":"ContainerStarted","Data":"12d3b8949ca55b2e9a347c0a483b12e4b7b21c7794ef16ca85fe941ce45e015e"} Jul 22 19:33:52.775781 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:52.775747 2583 generic.go:356] "Generic (PLEG): container finished" podID="b01a4fa3-4595-4c77-a6e9-82278e0bf264" containerID="c60f4be015a2cdd9bc36c468de949dacc85dba0aa7ad07ab1d1cb77007f7fc59" exitCode=0 Jul 22 19:33:52.776191 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:52.775829 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q" event={"ID":"b01a4fa3-4595-4c77-a6e9-82278e0bf264","Type":"ContainerDied","Data":"c60f4be015a2cdd9bc36c468de949dacc85dba0aa7ad07ab1d1cb77007f7fc59"} Jul 22 19:33:53.923041 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:53.923018 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q" Jul 22 19:33:53.987287 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:53.987253 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01a4fa3-4595-4c77-a6e9-82278e0bf264-maas-api-service-ca\") pod \"b01a4fa3-4595-4c77-a6e9-82278e0bf264\" (UID: \"b01a4fa3-4595-4c77-a6e9-82278e0bf264\") " Jul 22 19:33:53.987633 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:53.987611 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01a4fa3-4595-4c77-a6e9-82278e0bf264-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b01a4fa3-4595-4c77-a6e9-82278e0bf264" (UID: "b01a4fa3-4595-4c77-a6e9-82278e0bf264"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:33:54.089046 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:54.088957 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01a4fa3-4595-4c77-a6e9-82278e0bf264-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:33:54.790581 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:54.790550 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q" Jul 22 19:33:54.790753 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:54.790558 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q" event={"ID":"b01a4fa3-4595-4c77-a6e9-82278e0bf264","Type":"ContainerDied","Data":"12d3b8949ca55b2e9a347c0a483b12e4b7b21c7794ef16ca85fe941ce45e015e"} Jul 22 19:33:54.790753 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:33:54.790672 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12d3b8949ca55b2e9a347c0a483b12e4b7b21c7794ef16ca85fe941ce45e015e" Jul 22 19:34:01.483696 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:01.483612 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6"] Jul 22 19:34:01.484417 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:01.484380 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" podUID="ec1e9e4b-f81b-40d8-b85f-19e549ad95de" containerName="maas-api" containerID="cri-o://81066b7652a447e6c998a6e1547e359dba3d3c7f4ae83a00c75d1e8164e76f99" gracePeriod=30 Jul 22 19:34:01.835375 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:01.835342 2583 generic.go:356] "Generic (PLEG): container finished" podID="ec1e9e4b-f81b-40d8-b85f-19e549ad95de" containerID="81066b7652a447e6c998a6e1547e359dba3d3c7f4ae83a00c75d1e8164e76f99" exitCode=0 Jul 22 19:34:01.835551 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:01.835420 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" event={"ID":"ec1e9e4b-f81b-40d8-b85f-19e549ad95de","Type":"ContainerDied","Data":"81066b7652a447e6c998a6e1547e359dba3d3c7f4ae83a00c75d1e8164e76f99"} Jul 22 19:34:01.952362 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:01.952337 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" Jul 22 19:34:02.066675 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:02.066644 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-openshift-service-ca\") pod \"ec1e9e4b-f81b-40d8-b85f-19e549ad95de\" (UID: \"ec1e9e4b-f81b-40d8-b85f-19e549ad95de\") " Jul 22 19:34:02.066892 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:02.066767 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-maas-api-tls\") pod \"ec1e9e4b-f81b-40d8-b85f-19e549ad95de\" (UID: \"ec1e9e4b-f81b-40d8-b85f-19e549ad95de\") " Jul 22 19:34:02.066892 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:02.066804 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c8dq\" (UniqueName: \"kubernetes.io/projected/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-kube-api-access-9c8dq\") pod \"ec1e9e4b-f81b-40d8-b85f-19e549ad95de\" (UID: \"ec1e9e4b-f81b-40d8-b85f-19e549ad95de\") " Jul 22 19:34:02.067100 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:02.067078 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec1e9e4b-f81b-40d8-b85f-19e549ad95de" (UID: "ec1e9e4b-f81b-40d8-b85f-19e549ad95de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:34:02.067228 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:02.067201 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-openshift-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:34:02.069018 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:02.068994 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec1e9e4b-f81b-40d8-b85f-19e549ad95de" (UID: "ec1e9e4b-f81b-40d8-b85f-19e549ad95de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:34:02.069111 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:02.069056 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-kube-api-access-9c8dq" (OuterVolumeSpecName: "kube-api-access-9c8dq") pod "ec1e9e4b-f81b-40d8-b85f-19e549ad95de" (UID: "ec1e9e4b-f81b-40d8-b85f-19e549ad95de"). InnerVolumeSpecName "kube-api-access-9c8dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:34:02.168299 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:02.168265 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-maas-api-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:34:02.168299 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:02.168293 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c8dq\" (UniqueName: \"kubernetes.io/projected/ec1e9e4b-f81b-40d8-b85f-19e549ad95de-kube-api-access-9c8dq\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:34:02.843840 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:02.843811 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" Jul 22 19:34:02.844277 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:02.843816 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6" event={"ID":"ec1e9e4b-f81b-40d8-b85f-19e549ad95de","Type":"ContainerDied","Data":"b7fdc54076df5d3782b7efac6241eaf6d6624ef98ff57f36c10febea33e87c74"} Jul 22 19:34:02.844277 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:02.843953 2583 scope.go:117] "RemoveContainer" containerID="81066b7652a447e6c998a6e1547e359dba3d3c7f4ae83a00c75d1e8164e76f99" Jul 22 19:34:02.868006 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:02.867972 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6"] Jul 22 19:34:02.872539 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:02.872513 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-735d50-f77f6df88-825s6"] Jul 22 19:34:03.550688 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:03.550654 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec1e9e4b-f81b-40d8-b85f-19e549ad95de" path="/var/lib/kubelet/pods/ec1e9e4b-f81b-40d8-b85f-19e549ad95de/volumes" Jul 22 19:34:06.194097 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:06.194055 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q"] Jul 22 19:34:06.196924 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:06.196896 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-735d50-rpc4q"] Jul 22 19:34:07.551389 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:07.551349 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b01a4fa3-4595-4c77-a6e9-82278e0bf264" path="/var/lib/kubelet/pods/b01a4fa3-4595-4c77-a6e9-82278e0bf264/volumes" Jul 22 19:34:12.840655 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:12.840618 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv"] Jul 22 19:34:12.843034 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:12.841089 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01a4fa3-4595-4c77-a6e9-82278e0bf264" containerName="revoke-keys" Jul 22 19:34:12.843034 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:12.841100 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01a4fa3-4595-4c77-a6e9-82278e0bf264" containerName="revoke-keys" Jul 22 19:34:12.843034 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:12.841114 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec1e9e4b-f81b-40d8-b85f-19e549ad95de" containerName="maas-api" Jul 22 19:34:12.843034 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:12.841120 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec1e9e4b-f81b-40d8-b85f-19e549ad95de" containerName="maas-api" Jul 22 19:34:12.843034 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:12.841231 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec1e9e4b-f81b-40d8-b85f-19e549ad95de" containerName="maas-api" Jul 22 19:34:12.843034 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:12.841240 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b01a4fa3-4595-4c77-a6e9-82278e0bf264" containerName="revoke-keys" Jul 22 19:34:12.844333 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:12.844317 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv" Jul 22 19:34:12.846752 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:12.846728 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jhv4j\"" Jul 22 19:34:12.851764 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:12.851736 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv"] Jul 22 19:34:12.976993 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:12.976936 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76065637-aa81-4c11-86c6-a2502c080c6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv\" (UID: \"76065637-aa81-4c11-86c6-a2502c080c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv" Jul 22 19:34:13.078270 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:13.078231 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76065637-aa81-4c11-86c6-a2502c080c6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv\" (UID: \"76065637-aa81-4c11-86c6-a2502c080c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv" Jul 22 19:34:13.078888 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:13.078868 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76065637-aa81-4c11-86c6-a2502c080c6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv\" (UID: \"76065637-aa81-4c11-86c6-a2502c080c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv" Jul 22 19:34:13.156538 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:13.156412 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv" Jul 22 19:34:13.302246 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:13.302218 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv"] Jul 22 19:34:13.303618 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:34:13.303586 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76065637_aa81_4c11_86c6_a2502c080c6e.slice/crio-94870c882d13d619fc8c8024e6e7eee73814187474dfcd529b94d7beb8087186 WatchSource:0}: Error finding container 94870c882d13d619fc8c8024e6e7eee73814187474dfcd529b94d7beb8087186: Status 404 returned error can't find the container with id 94870c882d13d619fc8c8024e6e7eee73814187474dfcd529b94d7beb8087186 Jul 22 19:34:13.919099 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:13.919057 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv" event={"ID":"76065637-aa81-4c11-86c6-a2502c080c6e","Type":"ContainerStarted","Data":"dd5ce1f766f5e9b423521b864a89a08f3236792778407091cfa4bf9646cc64b4"} Jul 22 19:34:13.919544 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:13.919121 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv" event={"ID":"76065637-aa81-4c11-86c6-a2502c080c6e","Type":"ContainerStarted","Data":"94870c882d13d619fc8c8024e6e7eee73814187474dfcd529b94d7beb8087186"} Jul 22 19:34:13.936119 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:13.936066 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv" podStartSLOduration=1.9360515839999999 podStartE2EDuration="1.936051584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:34:13.933630753 +0000 UTC m=+2256.845794834" watchObservedRunningTime="2026-07-22 19:34:13.936051584 +0000 UTC m=+2256.848215678" Jul 22 19:34:14.927222 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:14.927185 2583 generic.go:356] "Generic (PLEG): container finished" podID="76065637-aa81-4c11-86c6-a2502c080c6e" containerID="dd5ce1f766f5e9b423521b864a89a08f3236792778407091cfa4bf9646cc64b4" exitCode=0 Jul 22 19:34:14.927607 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:14.927233 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv" event={"ID":"76065637-aa81-4c11-86c6-a2502c080c6e","Type":"ContainerDied","Data":"dd5ce1f766f5e9b423521b864a89a08f3236792778407091cfa4bf9646cc64b4"} Jul 22 19:34:16.074596 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:16.074552 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv" Jul 22 19:34:16.211029 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:16.210940 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76065637-aa81-4c11-86c6-a2502c080c6e-maas-api-service-ca\") pod \"76065637-aa81-4c11-86c6-a2502c080c6e\" (UID: \"76065637-aa81-4c11-86c6-a2502c080c6e\") " Jul 22 19:34:16.211312 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:16.211287 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76065637-aa81-4c11-86c6-a2502c080c6e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "76065637-aa81-4c11-86c6-a2502c080c6e" (UID: "76065637-aa81-4c11-86c6-a2502c080c6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:34:16.312081 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:16.312032 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76065637-aa81-4c11-86c6-a2502c080c6e-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:34:16.939941 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:16.939908 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv" Jul 22 19:34:16.939941 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:16.939936 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv" event={"ID":"76065637-aa81-4c11-86c6-a2502c080c6e","Type":"ContainerDied","Data":"94870c882d13d619fc8c8024e6e7eee73814187474dfcd529b94d7beb8087186"} Jul 22 19:34:16.940207 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:16.940005 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94870c882d13d619fc8c8024e6e7eee73814187474dfcd529b94d7beb8087186" Jul 22 19:34:23.230666 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.230553 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm"] Jul 22 19:34:23.231177 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.230906 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" podUID="68be4238-55c0-43e9-aa06-a7ef5e3d6ca2" containerName="maas-api" containerID="cri-o://078baae562924767e651c3ac5d41ae90c11a3a671cbef239e9fa80df7e5e29e4" gracePeriod=30 Jul 22 19:34:23.498316 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.498247 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" Jul 22 19:34:23.581130 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.581096 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-openshift-service-ca\") pod \"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2\" (UID: \"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2\") " Jul 22 19:34:23.581285 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.581149 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68fm6\" (UniqueName: \"kubernetes.io/projected/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-kube-api-access-68fm6\") pod \"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2\" (UID: \"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2\") " Jul 22 19:34:23.581285 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.581250 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-maas-api-tls\") pod \"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2\" (UID: \"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2\") " Jul 22 19:34:23.581584 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.581552 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68be4238-55c0-43e9-aa06-a7ef5e3d6ca2" (UID: "68be4238-55c0-43e9-aa06-a7ef5e3d6ca2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:34:23.583475 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.583447 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68be4238-55c0-43e9-aa06-a7ef5e3d6ca2" (UID: "68be4238-55c0-43e9-aa06-a7ef5e3d6ca2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:34:23.583475 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.583459 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-kube-api-access-68fm6" (OuterVolumeSpecName: "kube-api-access-68fm6") pod "68be4238-55c0-43e9-aa06-a7ef5e3d6ca2" (UID: "68be4238-55c0-43e9-aa06-a7ef5e3d6ca2"). InnerVolumeSpecName "kube-api-access-68fm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:34:23.682290 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.682254 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-maas-api-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:34:23.682290 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.682284 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-openshift-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:34:23.682290 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.682294 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68fm6\" (UniqueName: \"kubernetes.io/projected/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2-kube-api-access-68fm6\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:34:23.983270 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.983235 2583 generic.go:356] "Generic (PLEG): container finished" podID="68be4238-55c0-43e9-aa06-a7ef5e3d6ca2" containerID="078baae562924767e651c3ac5d41ae90c11a3a671cbef239e9fa80df7e5e29e4" exitCode=0 Jul 22 19:34:23.983421 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.983275 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" Jul 22 19:34:23.983421 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.983325 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" event={"ID":"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2","Type":"ContainerDied","Data":"078baae562924767e651c3ac5d41ae90c11a3a671cbef239e9fa80df7e5e29e4"} Jul 22 19:34:23.983421 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.983377 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm" event={"ID":"68be4238-55c0-43e9-aa06-a7ef5e3d6ca2","Type":"ContainerDied","Data":"839caad6e594d034990c7abd836eb0699bb4c6a8f2bb2ffc1c5dd36ac35deecc"} Jul 22 19:34:23.983421 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.983391 2583 scope.go:117] "RemoveContainer" containerID="078baae562924767e651c3ac5d41ae90c11a3a671cbef239e9fa80df7e5e29e4" Jul 22 19:34:23.994264 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.994240 2583 scope.go:117] "RemoveContainer" containerID="078baae562924767e651c3ac5d41ae90c11a3a671cbef239e9fa80df7e5e29e4" Jul 22 19:34:23.994545 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:34:23.994527 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078baae562924767e651c3ac5d41ae90c11a3a671cbef239e9fa80df7e5e29e4\": container with ID starting with 078baae562924767e651c3ac5d41ae90c11a3a671cbef239e9fa80df7e5e29e4 not found: ID does not exist" containerID="078baae562924767e651c3ac5d41ae90c11a3a671cbef239e9fa80df7e5e29e4" Jul 22 19:34:23.994622 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:23.994553 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078baae562924767e651c3ac5d41ae90c11a3a671cbef239e9fa80df7e5e29e4"} err="failed to get container status \"078baae562924767e651c3ac5d41ae90c11a3a671cbef239e9fa80df7e5e29e4\": rpc error: code = NotFound desc = could not find container \"078baae562924767e651c3ac5d41ae90c11a3a671cbef239e9fa80df7e5e29e4\": container with ID starting with 078baae562924767e651c3ac5d41ae90c11a3a671cbef239e9fa80df7e5e29e4 not found: ID does not exist" Jul 22 19:34:24.006330 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:24.006302 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm"] Jul 22 19:34:24.009047 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:24.009022 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc2828-6d6d9dfdb4-zjnjm"] Jul 22 19:34:25.551118 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:25.551081 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68be4238-55c0-43e9-aa06-a7ef5e3d6ca2" path="/var/lib/kubelet/pods/68be4238-55c0-43e9-aa06-a7ef5e3d6ca2/volumes" Jul 22 19:34:27.924756 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:27.924679 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv"] Jul 22 19:34:27.930654 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:27.930612 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc2828-5vrrv"] Jul 22 19:34:29.551087 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:29.551054 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76065637-aa81-4c11-86c6-a2502c080c6e" path="/var/lib/kubelet/pods/76065637-aa81-4c11-86c6-a2502c080c6e/volumes" Jul 22 19:34:37.912321 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:37.912287 2583 scope.go:117] "RemoveContainer" containerID="efaea5733f66cd2fe6a148f406ed315433921109997d5f119801be54568df469" Jul 22 19:34:38.581987 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.581953 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j"] Jul 22 19:34:38.582413 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.582399 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76065637-aa81-4c11-86c6-a2502c080c6e" containerName="revoke-keys" Jul 22 19:34:38.582459 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.582414 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="76065637-aa81-4c11-86c6-a2502c080c6e" containerName="revoke-keys" Jul 22 19:34:38.582459 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.582435 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68be4238-55c0-43e9-aa06-a7ef5e3d6ca2" containerName="maas-api" Jul 22 19:34:38.582459 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.582440 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="68be4238-55c0-43e9-aa06-a7ef5e3d6ca2" containerName="maas-api" Jul 22 19:34:38.582583 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.582536 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="76065637-aa81-4c11-86c6-a2502c080c6e" containerName="revoke-keys" Jul 22 19:34:38.582583 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.582545 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="68be4238-55c0-43e9-aa06-a7ef5e3d6ca2" containerName="maas-api" Jul 22 19:34:38.587310 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.587282 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" Jul 22 19:34:38.589915 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.589888 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3c150b-serving-cert\"" Jul 22 19:34:38.590076 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.590049 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t4d8t\"" Jul 22 19:34:38.595216 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.595190 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j"] Jul 22 19:34:38.624811 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.624776 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454f0757-e734-463d-bc50-88de4bf19da7-openshift-service-ca\") pod \"maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j\" (UID: \"454f0757-e734-463d-bc50-88de4bf19da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" Jul 22 19:34:38.625000 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.624863 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvsxx\" (UniqueName: \"kubernetes.io/projected/454f0757-e734-463d-bc50-88de4bf19da7-kube-api-access-zvsxx\") pod \"maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j\" (UID: \"454f0757-e734-463d-bc50-88de4bf19da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" Jul 22 19:34:38.625000 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.624911 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454f0757-e734-463d-bc50-88de4bf19da7-maas-api-tls\") pod \"maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j\" (UID: \"454f0757-e734-463d-bc50-88de4bf19da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" Jul 22 19:34:38.725460 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.725420 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvsxx\" (UniqueName: \"kubernetes.io/projected/454f0757-e734-463d-bc50-88de4bf19da7-kube-api-access-zvsxx\") pod \"maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j\" (UID: \"454f0757-e734-463d-bc50-88de4bf19da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" Jul 22 19:34:38.725665 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.725477 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454f0757-e734-463d-bc50-88de4bf19da7-maas-api-tls\") pod \"maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j\" (UID: \"454f0757-e734-463d-bc50-88de4bf19da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" Jul 22 19:34:38.725665 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.725611 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454f0757-e734-463d-bc50-88de4bf19da7-openshift-service-ca\") pod \"maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j\" (UID: \"454f0757-e734-463d-bc50-88de4bf19da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" Jul 22 19:34:38.725778 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:34:38.725684 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-serving-cert: secret "maas-api-e2e-api-a-3c150b-serving-cert" not found Jul 22 19:34:38.725822 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:34:38.725789 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/454f0757-e734-463d-bc50-88de4bf19da7-maas-api-tls podName:454f0757-e734-463d-bc50-88de4bf19da7 nodeName:}" failed. No retries permitted until 2026-07-22 19:34:39.225754717 +0000 UTC m=+2282.137918765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/454f0757-e734-463d-bc50-88de4bf19da7-maas-api-tls") pod "maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" (UID: "454f0757-e734-463d-bc50-88de4bf19da7") : secret "maas-api-e2e-api-a-3c150b-serving-cert" not found Jul 22 19:34:38.726288 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.726271 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454f0757-e734-463d-bc50-88de4bf19da7-openshift-service-ca\") pod \"maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j\" (UID: \"454f0757-e734-463d-bc50-88de4bf19da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" Jul 22 19:34:38.737649 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:38.737617 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvsxx\" (UniqueName: \"kubernetes.io/projected/454f0757-e734-463d-bc50-88de4bf19da7-kube-api-access-zvsxx\") pod \"maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j\" (UID: \"454f0757-e734-463d-bc50-88de4bf19da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" Jul 22 19:34:39.230059 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:39.230025 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454f0757-e734-463d-bc50-88de4bf19da7-maas-api-tls\") pod \"maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j\" (UID: \"454f0757-e734-463d-bc50-88de4bf19da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" Jul 22 19:34:39.232674 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:39.232644 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454f0757-e734-463d-bc50-88de4bf19da7-maas-api-tls\") pod \"maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j\" (UID: \"454f0757-e734-463d-bc50-88de4bf19da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" Jul 22 19:34:39.501169 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:39.501064 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" Jul 22 19:34:39.651923 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:39.651894 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j"] Jul 22 19:34:39.653705 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:34:39.653677 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454f0757_e734_463d_bc50_88de4bf19da7.slice/crio-3b5558caf19cbe173d673518de8a3e4d724e261ed48a4dadcb961c33d9cbe371 WatchSource:0}: Error finding container 3b5558caf19cbe173d673518de8a3e4d724e261ed48a4dadcb961c33d9cbe371: Status 404 returned error can't find the container with id 3b5558caf19cbe173d673518de8a3e4d724e261ed48a4dadcb961c33d9cbe371 Jul 22 19:34:39.655138 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:39.655118 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 19:34:40.087019 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:40.086977 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" event={"ID":"454f0757-e734-463d-bc50-88de4bf19da7","Type":"ContainerStarted","Data":"3b5558caf19cbe173d673518de8a3e4d724e261ed48a4dadcb961c33d9cbe371"} Jul 22 19:34:41.095630 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:41.095592 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" event={"ID":"454f0757-e734-463d-bc50-88de4bf19da7","Type":"ContainerStarted","Data":"5cd340abcd23f5865b25dcfd051333ccf9d80b79dece58446a5bed00027844fb"} Jul 22 19:34:41.096067 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:41.095703 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" Jul 22 19:34:41.113756 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:41.113702 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" podStartSLOduration=2.474988841 podStartE2EDuration="3.113685394s" totalImagesPullingTime="638.696553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:34:41.110356801 +0000 UTC m=+2284.022520883" watchObservedRunningTime="2026-07-22 19:34:41.113685394 +0000 UTC m=+2284.025849461" Jul 22 19:34:46.734925 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:46.734886 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj"] Jul 22 19:34:46.739613 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:46.739586 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" Jul 22 19:34:46.742120 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:46.742098 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b3fa31-serving-cert\"" Jul 22 19:34:46.749690 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:46.749545 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj"] Jul 22 19:34:46.801644 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:46.801604 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlg5d\" (UniqueName: \"kubernetes.io/projected/a220129c-d357-426b-8862-597e121f97e4-kube-api-access-tlg5d\") pod \"maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj\" (UID: \"a220129c-d357-426b-8862-597e121f97e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" Jul 22 19:34:46.801826 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:46.801651 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a220129c-d357-426b-8862-597e121f97e4-maas-api-tls\") pod \"maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj\" (UID: \"a220129c-d357-426b-8862-597e121f97e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" Jul 22 19:34:46.801826 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:46.801733 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a220129c-d357-426b-8862-597e121f97e4-openshift-service-ca\") pod \"maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj\" (UID: \"a220129c-d357-426b-8862-597e121f97e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" Jul 22 19:34:46.903104 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:46.903065 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a220129c-d357-426b-8862-597e121f97e4-openshift-service-ca\") pod \"maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj\" (UID: \"a220129c-d357-426b-8862-597e121f97e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" Jul 22 19:34:46.903280 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:46.903131 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlg5d\" (UniqueName: \"kubernetes.io/projected/a220129c-d357-426b-8862-597e121f97e4-kube-api-access-tlg5d\") pod \"maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj\" (UID: \"a220129c-d357-426b-8862-597e121f97e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" Jul 22 19:34:46.903280 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:46.903154 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a220129c-d357-426b-8862-597e121f97e4-maas-api-tls\") pod \"maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj\" (UID: \"a220129c-d357-426b-8862-597e121f97e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" Jul 22 19:34:46.903763 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:46.903739 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a220129c-d357-426b-8862-597e121f97e4-openshift-service-ca\") pod \"maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj\" (UID: \"a220129c-d357-426b-8862-597e121f97e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" Jul 22 19:34:46.905803 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:46.905781 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a220129c-d357-426b-8862-597e121f97e4-maas-api-tls\") pod \"maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj\" (UID: \"a220129c-d357-426b-8862-597e121f97e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" Jul 22 19:34:46.912309 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:46.912281 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlg5d\" (UniqueName: \"kubernetes.io/projected/a220129c-d357-426b-8862-597e121f97e4-kube-api-access-tlg5d\") pod \"maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj\" (UID: \"a220129c-d357-426b-8862-597e121f97e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" Jul 22 19:34:47.062788 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:47.062752 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" Jul 22 19:34:47.108821 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:47.108793 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" Jul 22 19:34:47.225113 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:47.225072 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj"] Jul 22 19:34:47.226178 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:34:47.226155 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda220129c_d357_426b_8862_597e121f97e4.slice/crio-64fcf5910b6f7efb327a81224eb13cae1c8c68bd3c2a9daa97523415d6b2e533 WatchSource:0}: Error finding container 64fcf5910b6f7efb327a81224eb13cae1c8c68bd3c2a9daa97523415d6b2e533: Status 404 returned error can't find the container with id 64fcf5910b6f7efb327a81224eb13cae1c8c68bd3c2a9daa97523415d6b2e533 Jul 22 19:34:48.159858 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:48.159820 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" event={"ID":"a220129c-d357-426b-8862-597e121f97e4","Type":"ContainerStarted","Data":"3b0ac7b6c86e40b619883dd7b9a99ee0caf05337f94df2090286ce3f20afcbcd"} Jul 22 19:34:48.160254 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:48.159873 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" event={"ID":"a220129c-d357-426b-8862-597e121f97e4","Type":"ContainerStarted","Data":"64fcf5910b6f7efb327a81224eb13cae1c8c68bd3c2a9daa97523415d6b2e533"} Jul 22 19:34:48.160254 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:48.159894 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" Jul 22 19:34:48.190815 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:48.190756 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" podStartSLOduration=1.7681051779999999 podStartE2EDuration="2.190739571s" totalImagesPullingTime="422.634393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:34:48.188475688 +0000 UTC m=+2291.100639756" watchObservedRunningTime="2026-07-22 19:34:48.190739571 +0000 UTC m=+2291.102903639" Jul 22 19:34:54.172859 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:54.172829 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" Jul 22 19:34:58.999244 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:58.999198 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf"] Jul 22 19:34:59.004241 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:59.004217 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf" Jul 22 19:34:59.006784 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:59.006758 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jhv4j\"" Jul 22 19:34:59.009901 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:59.009874 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf"] Jul 22 19:34:59.128189 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:59.128152 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c01fbee1-f36a-45c5-b467-34954636eefc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf\" (UID: \"c01fbee1-f36a-45c5-b467-34954636eefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf" Jul 22 19:34:59.229104 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:59.229073 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c01fbee1-f36a-45c5-b467-34954636eefc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf\" (UID: \"c01fbee1-f36a-45c5-b467-34954636eefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf" Jul 22 19:34:59.229755 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:59.229737 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c01fbee1-f36a-45c5-b467-34954636eefc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf\" (UID: \"c01fbee1-f36a-45c5-b467-34954636eefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf" Jul 22 19:34:59.317819 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:59.317774 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf" Jul 22 19:34:59.478020 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:34:59.477955 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf"] Jul 22 19:34:59.479302 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:34:59.479274 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01fbee1_f36a_45c5_b467_34954636eefc.slice/crio-767975a35234cd2803a655f0b2781b91efbad33301ebeb6e1eaf79d54eb71dd5 WatchSource:0}: Error finding container 767975a35234cd2803a655f0b2781b91efbad33301ebeb6e1eaf79d54eb71dd5: Status 404 returned error can't find the container with id 767975a35234cd2803a655f0b2781b91efbad33301ebeb6e1eaf79d54eb71dd5 Jul 22 19:35:00.239192 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:00.239158 2583 generic.go:356] "Generic (PLEG): container finished" podID="c01fbee1-f36a-45c5-b467-34954636eefc" containerID="26b8cb36773a2f1e8f30aaf3a016d66028c4c76792c99a92b1d1635047ec4925" exitCode=0 Jul 22 19:35:00.239593 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:00.239197 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf" event={"ID":"c01fbee1-f36a-45c5-b467-34954636eefc","Type":"ContainerDied","Data":"26b8cb36773a2f1e8f30aaf3a016d66028c4c76792c99a92b1d1635047ec4925"} Jul 22 19:35:00.239593 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:00.239249 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf" event={"ID":"c01fbee1-f36a-45c5-b467-34954636eefc","Type":"ContainerStarted","Data":"767975a35234cd2803a655f0b2781b91efbad33301ebeb6e1eaf79d54eb71dd5"} Jul 22 19:35:01.384021 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:01.383996 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf" Jul 22 19:35:01.551475 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:01.551442 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c01fbee1-f36a-45c5-b467-34954636eefc-maas-api-service-ca\") pod \"c01fbee1-f36a-45c5-b467-34954636eefc\" (UID: \"c01fbee1-f36a-45c5-b467-34954636eefc\") " Jul 22 19:35:01.551845 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:01.551823 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01fbee1-f36a-45c5-b467-34954636eefc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c01fbee1-f36a-45c5-b467-34954636eefc" (UID: "c01fbee1-f36a-45c5-b467-34954636eefc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:35:01.653262 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:01.653211 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c01fbee1-f36a-45c5-b467-34954636eefc-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:35:02.253729 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:02.253703 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf" Jul 22 19:35:02.253910 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:02.253701 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf" event={"ID":"c01fbee1-f36a-45c5-b467-34954636eefc","Type":"ContainerDied","Data":"767975a35234cd2803a655f0b2781b91efbad33301ebeb6e1eaf79d54eb71dd5"} Jul 22 19:35:02.253910 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:02.253830 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="767975a35234cd2803a655f0b2781b91efbad33301ebeb6e1eaf79d54eb71dd5" Jul 22 19:35:14.994404 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:14.994366 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j"] Jul 22 19:35:14.995044 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:14.994846 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" podUID="454f0757-e734-463d-bc50-88de4bf19da7" containerName="maas-api" containerID="cri-o://5cd340abcd23f5865b25dcfd051333ccf9d80b79dece58446a5bed00027844fb" gracePeriod=30 Jul 22 19:35:15.249974 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.249909 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" Jul 22 19:35:15.282631 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.282595 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454f0757-e734-463d-bc50-88de4bf19da7-openshift-service-ca\") pod \"454f0757-e734-463d-bc50-88de4bf19da7\" (UID: \"454f0757-e734-463d-bc50-88de4bf19da7\") " Jul 22 19:35:15.282815 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.282727 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454f0757-e734-463d-bc50-88de4bf19da7-maas-api-tls\") pod \"454f0757-e734-463d-bc50-88de4bf19da7\" (UID: \"454f0757-e734-463d-bc50-88de4bf19da7\") " Jul 22 19:35:15.282815 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.282772 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvsxx\" (UniqueName: \"kubernetes.io/projected/454f0757-e734-463d-bc50-88de4bf19da7-kube-api-access-zvsxx\") pod \"454f0757-e734-463d-bc50-88de4bf19da7\" (UID: \"454f0757-e734-463d-bc50-88de4bf19da7\") " Jul 22 19:35:15.283011 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.282974 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454f0757-e734-463d-bc50-88de4bf19da7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "454f0757-e734-463d-bc50-88de4bf19da7" (UID: "454f0757-e734-463d-bc50-88de4bf19da7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:35:15.283140 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.283121 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454f0757-e734-463d-bc50-88de4bf19da7-openshift-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:35:15.285093 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.285065 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454f0757-e734-463d-bc50-88de4bf19da7-kube-api-access-zvsxx" (OuterVolumeSpecName: "kube-api-access-zvsxx") pod "454f0757-e734-463d-bc50-88de4bf19da7" (UID: "454f0757-e734-463d-bc50-88de4bf19da7"). InnerVolumeSpecName "kube-api-access-zvsxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:35:15.285415 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.285394 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/454f0757-e734-463d-bc50-88de4bf19da7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "454f0757-e734-463d-bc50-88de4bf19da7" (UID: "454f0757-e734-463d-bc50-88de4bf19da7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:35:15.346654 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.346610 2583 generic.go:356] "Generic (PLEG): container finished" podID="454f0757-e734-463d-bc50-88de4bf19da7" containerID="5cd340abcd23f5865b25dcfd051333ccf9d80b79dece58446a5bed00027844fb" exitCode=0 Jul 22 19:35:15.346654 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.346660 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" Jul 22 19:35:15.346866 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.346699 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" event={"ID":"454f0757-e734-463d-bc50-88de4bf19da7","Type":"ContainerDied","Data":"5cd340abcd23f5865b25dcfd051333ccf9d80b79dece58446a5bed00027844fb"} Jul 22 19:35:15.346866 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.346778 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j" event={"ID":"454f0757-e734-463d-bc50-88de4bf19da7","Type":"ContainerDied","Data":"3b5558caf19cbe173d673518de8a3e4d724e261ed48a4dadcb961c33d9cbe371"} Jul 22 19:35:15.346866 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.346801 2583 scope.go:117] "RemoveContainer" containerID="5cd340abcd23f5865b25dcfd051333ccf9d80b79dece58446a5bed00027844fb" Jul 22 19:35:15.357376 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.357354 2583 scope.go:117] "RemoveContainer" containerID="5cd340abcd23f5865b25dcfd051333ccf9d80b79dece58446a5bed00027844fb" Jul 22 19:35:15.357670 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:35:15.357647 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd340abcd23f5865b25dcfd051333ccf9d80b79dece58446a5bed00027844fb\": container with ID starting with 5cd340abcd23f5865b25dcfd051333ccf9d80b79dece58446a5bed00027844fb not found: ID does not exist" containerID="5cd340abcd23f5865b25dcfd051333ccf9d80b79dece58446a5bed00027844fb" Jul 22 19:35:15.357767 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.357678 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd340abcd23f5865b25dcfd051333ccf9d80b79dece58446a5bed00027844fb"} err="failed to get container status \"5cd340abcd23f5865b25dcfd051333ccf9d80b79dece58446a5bed00027844fb\": rpc error: code = NotFound desc = could not find container \"5cd340abcd23f5865b25dcfd051333ccf9d80b79dece58446a5bed00027844fb\": container with ID starting with 5cd340abcd23f5865b25dcfd051333ccf9d80b79dece58446a5bed00027844fb not found: ID does not exist" Jul 22 19:35:15.368941 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.368814 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j"] Jul 22 19:35:15.371839 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.371804 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c150b-67c9fdb57-snk2j"] Jul 22 19:35:15.384822 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.384793 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454f0757-e734-463d-bc50-88de4bf19da7-maas-api-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:35:15.384927 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.384825 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvsxx\" (UniqueName: \"kubernetes.io/projected/454f0757-e734-463d-bc50-88de4bf19da7-kube-api-access-zvsxx\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:35:15.551663 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:15.551629 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="454f0757-e734-463d-bc50-88de4bf19da7" path="/var/lib/kubelet/pods/454f0757-e734-463d-bc50-88de4bf19da7/volumes" Jul 22 19:35:19.074400 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:19.074365 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf"] Jul 22 19:35:19.076956 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:19.076933 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c150b-2cpdf"] Jul 22 19:35:19.551343 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:19.551311 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c01fbee1-f36a-45c5-b467-34954636eefc" path="/var/lib/kubelet/pods/c01fbee1-f36a-45c5-b467-34954636eefc/volumes" Jul 22 19:35:25.900028 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:25.899990 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg"] Jul 22 19:35:25.900701 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:25.900679 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454f0757-e734-463d-bc50-88de4bf19da7" containerName="maas-api" Jul 22 19:35:25.900793 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:25.900703 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="454f0757-e734-463d-bc50-88de4bf19da7" containerName="maas-api" Jul 22 19:35:25.900793 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:25.900714 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01fbee1-f36a-45c5-b467-34954636eefc" containerName="revoke-keys" Jul 22 19:35:25.900793 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:25.900721 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01fbee1-f36a-45c5-b467-34954636eefc" containerName="revoke-keys" Jul 22 19:35:25.900947 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:25.900844 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="454f0757-e734-463d-bc50-88de4bf19da7" containerName="maas-api" Jul 22 19:35:25.900947 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:25.900855 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01fbee1-f36a-45c5-b467-34954636eefc" containerName="revoke-keys" Jul 22 19:35:25.904098 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:25.904078 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg" Jul 22 19:35:25.906528 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:25.906485 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jhv4j\"" Jul 22 19:35:25.911769 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:25.911743 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg"] Jul 22 19:35:25.990319 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:25.990279 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b009f5d1-f274-4c2b-b982-dee064e32e45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg\" (UID: \"b009f5d1-f274-4c2b-b982-dee064e32e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg" Jul 22 19:35:26.091567 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:26.091526 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b009f5d1-f274-4c2b-b982-dee064e32e45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg\" (UID: \"b009f5d1-f274-4c2b-b982-dee064e32e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg" Jul 22 19:35:26.092107 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:26.092085 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b009f5d1-f274-4c2b-b982-dee064e32e45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg\" (UID: \"b009f5d1-f274-4c2b-b982-dee064e32e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg" Jul 22 19:35:26.217687 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:26.217600 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg" Jul 22 19:35:26.369266 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:26.369234 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg"] Jul 22 19:35:26.370725 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:35:26.370698 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb009f5d1_f274_4c2b_b982_dee064e32e45.slice/crio-b2b60790c9852f5bf24ad66006a471cd2192625fa67489ab1ff3430ba7141d52 WatchSource:0}: Error finding container b2b60790c9852f5bf24ad66006a471cd2192625fa67489ab1ff3430ba7141d52: Status 404 returned error can't find the container with id b2b60790c9852f5bf24ad66006a471cd2192625fa67489ab1ff3430ba7141d52 Jul 22 19:35:26.416252 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:26.416217 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg" event={"ID":"b009f5d1-f274-4c2b-b982-dee064e32e45","Type":"ContainerStarted","Data":"b2b60790c9852f5bf24ad66006a471cd2192625fa67489ab1ff3430ba7141d52"} Jul 22 19:35:27.432281 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:27.432188 2583 generic.go:356] "Generic (PLEG): container finished" podID="b009f5d1-f274-4c2b-b982-dee064e32e45" containerID="55b46fb1864067723c6f7c3f3662a36ecc07435fb7cb831cc5e58a38a05e68fc" exitCode=0 Jul 22 19:35:27.432281 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:27.432219 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg" event={"ID":"b009f5d1-f274-4c2b-b982-dee064e32e45","Type":"ContainerDied","Data":"55b46fb1864067723c6f7c3f3662a36ecc07435fb7cb831cc5e58a38a05e68fc"} Jul 22 19:35:28.581044 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:28.581017 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg" Jul 22 19:35:28.716940 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:28.716842 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b009f5d1-f274-4c2b-b982-dee064e32e45-maas-api-service-ca\") pod \"b009f5d1-f274-4c2b-b982-dee064e32e45\" (UID: \"b009f5d1-f274-4c2b-b982-dee064e32e45\") " Jul 22 19:35:28.717279 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:28.717258 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b009f5d1-f274-4c2b-b982-dee064e32e45-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b009f5d1-f274-4c2b-b982-dee064e32e45" (UID: "b009f5d1-f274-4c2b-b982-dee064e32e45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:35:28.818321 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:28.818279 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b009f5d1-f274-4c2b-b982-dee064e32e45-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:35:29.447457 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:29.447425 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg" Jul 22 19:35:29.447650 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:29.447458 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg" event={"ID":"b009f5d1-f274-4c2b-b982-dee064e32e45","Type":"ContainerDied","Data":"b2b60790c9852f5bf24ad66006a471cd2192625fa67489ab1ff3430ba7141d52"} Jul 22 19:35:29.447650 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:29.447527 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2b60790c9852f5bf24ad66006a471cd2192625fa67489ab1ff3430ba7141d52" Jul 22 19:35:41.438164 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:41.438114 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj"] Jul 22 19:35:41.438666 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:41.438387 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" podUID="a220129c-d357-426b-8862-597e121f97e4" containerName="maas-api" containerID="cri-o://3b0ac7b6c86e40b619883dd7b9a99ee0caf05337f94df2090286ce3f20afcbcd" gracePeriod=30 Jul 22 19:35:41.702394 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:41.702333 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" Jul 22 19:35:41.846897 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:41.846864 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a220129c-d357-426b-8862-597e121f97e4-openshift-service-ca\") pod \"a220129c-d357-426b-8862-597e121f97e4\" (UID: \"a220129c-d357-426b-8862-597e121f97e4\") " Jul 22 19:35:41.847080 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:41.847001 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a220129c-d357-426b-8862-597e121f97e4-maas-api-tls\") pod \"a220129c-d357-426b-8862-597e121f97e4\" (UID: \"a220129c-d357-426b-8862-597e121f97e4\") " Jul 22 19:35:41.847080 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:41.847037 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlg5d\" (UniqueName: \"kubernetes.io/projected/a220129c-d357-426b-8862-597e121f97e4-kube-api-access-tlg5d\") pod \"a220129c-d357-426b-8862-597e121f97e4\" (UID: \"a220129c-d357-426b-8862-597e121f97e4\") " Jul 22 19:35:41.847239 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:41.847215 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a220129c-d357-426b-8862-597e121f97e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a220129c-d357-426b-8862-597e121f97e4" (UID: "a220129c-d357-426b-8862-597e121f97e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:35:41.847322 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:41.847311 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a220129c-d357-426b-8862-597e121f97e4-openshift-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:35:41.849293 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:41.849265 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a220129c-d357-426b-8862-597e121f97e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a220129c-d357-426b-8862-597e121f97e4" (UID: "a220129c-d357-426b-8862-597e121f97e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:35:41.849293 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:41.849277 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a220129c-d357-426b-8862-597e121f97e4-kube-api-access-tlg5d" (OuterVolumeSpecName: "kube-api-access-tlg5d") pod "a220129c-d357-426b-8862-597e121f97e4" (UID: "a220129c-d357-426b-8862-597e121f97e4"). InnerVolumeSpecName "kube-api-access-tlg5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:35:41.948210 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:41.948166 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlg5d\" (UniqueName: \"kubernetes.io/projected/a220129c-d357-426b-8862-597e121f97e4-kube-api-access-tlg5d\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:35:41.948210 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:41.948202 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a220129c-d357-426b-8862-597e121f97e4-maas-api-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:35:42.533983 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:42.533944 2583 generic.go:356] "Generic (PLEG): container finished" podID="a220129c-d357-426b-8862-597e121f97e4" containerID="3b0ac7b6c86e40b619883dd7b9a99ee0caf05337f94df2090286ce3f20afcbcd" exitCode=0 Jul 22 19:35:42.534555 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:42.533989 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" Jul 22 19:35:42.534555 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:42.534005 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" event={"ID":"a220129c-d357-426b-8862-597e121f97e4","Type":"ContainerDied","Data":"3b0ac7b6c86e40b619883dd7b9a99ee0caf05337f94df2090286ce3f20afcbcd"} Jul 22 19:35:42.534555 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:42.534049 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj" event={"ID":"a220129c-d357-426b-8862-597e121f97e4","Type":"ContainerDied","Data":"64fcf5910b6f7efb327a81224eb13cae1c8c68bd3c2a9daa97523415d6b2e533"} Jul 22 19:35:42.534555 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:42.534070 2583 scope.go:117] "RemoveContainer" containerID="3b0ac7b6c86e40b619883dd7b9a99ee0caf05337f94df2090286ce3f20afcbcd" Jul 22 19:35:42.544372 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:42.544349 2583 scope.go:117] "RemoveContainer" containerID="3b0ac7b6c86e40b619883dd7b9a99ee0caf05337f94df2090286ce3f20afcbcd" Jul 22 19:35:42.544680 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:35:42.544659 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0ac7b6c86e40b619883dd7b9a99ee0caf05337f94df2090286ce3f20afcbcd\": container with ID starting with 3b0ac7b6c86e40b619883dd7b9a99ee0caf05337f94df2090286ce3f20afcbcd not found: ID does not exist" containerID="3b0ac7b6c86e40b619883dd7b9a99ee0caf05337f94df2090286ce3f20afcbcd" Jul 22 19:35:42.544731 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:42.544691 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0ac7b6c86e40b619883dd7b9a99ee0caf05337f94df2090286ce3f20afcbcd"} err="failed to get container status \"3b0ac7b6c86e40b619883dd7b9a99ee0caf05337f94df2090286ce3f20afcbcd\": rpc error: code = NotFound desc = could not find container \"3b0ac7b6c86e40b619883dd7b9a99ee0caf05337f94df2090286ce3f20afcbcd\": container with ID starting with 3b0ac7b6c86e40b619883dd7b9a99ee0caf05337f94df2090286ce3f20afcbcd not found: ID does not exist" Jul 22 19:35:42.556084 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:42.556051 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj"] Jul 22 19:35:42.559962 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:42.559935 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3fa31-7fb5675698-6xwdj"] Jul 22 19:35:43.550876 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:43.550842 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a220129c-d357-426b-8862-597e121f97e4" path="/var/lib/kubelet/pods/a220129c-d357-426b-8862-597e121f97e4/volumes" Jul 22 19:35:45.972350 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:45.972311 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg"] Jul 22 19:35:45.975401 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:45.975376 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3fa31-q9zdg"] Jul 22 19:35:47.551834 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:47.551801 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b009f5d1-f274-4c2b-b982-dee064e32e45" path="/var/lib/kubelet/pods/b009f5d1-f274-4c2b-b982-dee064e32e45/volumes" Jul 22 19:35:55.982558 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:55.982520 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc"] Jul 22 19:35:55.982971 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:55.982957 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a220129c-d357-426b-8862-597e121f97e4" containerName="maas-api" Jul 22 19:35:55.982971 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:55.982971 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a220129c-d357-426b-8862-597e121f97e4" containerName="maas-api" Jul 22 19:35:55.983053 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:55.982983 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b009f5d1-f274-4c2b-b982-dee064e32e45" containerName="revoke-keys" Jul 22 19:35:55.983053 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:55.982989 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b009f5d1-f274-4c2b-b982-dee064e32e45" containerName="revoke-keys" Jul 22 19:35:55.983116 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:55.983056 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a220129c-d357-426b-8862-597e121f97e4" containerName="maas-api" Jul 22 19:35:55.983116 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:55.983065 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b009f5d1-f274-4c2b-b982-dee064e32e45" containerName="revoke-keys" Jul 22 19:35:55.987729 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:55.987701 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" Jul 22 19:35:55.990584 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:55.990558 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-938311-serving-cert\"" Jul 22 19:35:55.990752 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:55.990669 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t4d8t\"" Jul 22 19:35:55.995060 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:55.995035 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc"] Jul 22 19:35:56.077755 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:56.077713 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9rb9\" (UniqueName: \"kubernetes.io/projected/54de62b5-5f44-4de5-9b7d-af353adc84a0-kube-api-access-x9rb9\") pod \"maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc\" (UID: \"54de62b5-5f44-4de5-9b7d-af353adc84a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" Jul 22 19:35:56.077932 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:56.077765 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54de62b5-5f44-4de5-9b7d-af353adc84a0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc\" (UID: \"54de62b5-5f44-4de5-9b7d-af353adc84a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" Jul 22 19:35:56.077932 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:56.077870 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54de62b5-5f44-4de5-9b7d-af353adc84a0-maas-api-tls\") pod \"maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc\" (UID: \"54de62b5-5f44-4de5-9b7d-af353adc84a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" Jul 22 19:35:56.178731 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:56.178697 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9rb9\" (UniqueName: \"kubernetes.io/projected/54de62b5-5f44-4de5-9b7d-af353adc84a0-kube-api-access-x9rb9\") pod \"maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc\" (UID: \"54de62b5-5f44-4de5-9b7d-af353adc84a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" Jul 22 19:35:56.178906 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:56.178740 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54de62b5-5f44-4de5-9b7d-af353adc84a0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc\" (UID: \"54de62b5-5f44-4de5-9b7d-af353adc84a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" Jul 22 19:35:56.178906 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:56.178797 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54de62b5-5f44-4de5-9b7d-af353adc84a0-maas-api-tls\") pod \"maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc\" (UID: \"54de62b5-5f44-4de5-9b7d-af353adc84a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" Jul 22 19:35:56.179435 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:56.179416 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54de62b5-5f44-4de5-9b7d-af353adc84a0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc\" (UID: \"54de62b5-5f44-4de5-9b7d-af353adc84a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" Jul 22 19:35:56.181298 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:56.181276 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54de62b5-5f44-4de5-9b7d-af353adc84a0-maas-api-tls\") pod \"maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc\" (UID: \"54de62b5-5f44-4de5-9b7d-af353adc84a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" Jul 22 19:35:56.192626 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:56.192598 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9rb9\" (UniqueName: \"kubernetes.io/projected/54de62b5-5f44-4de5-9b7d-af353adc84a0-kube-api-access-x9rb9\") pod \"maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc\" (UID: \"54de62b5-5f44-4de5-9b7d-af353adc84a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" Jul 22 19:35:56.300143 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:56.300103 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" Jul 22 19:35:56.457750 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:56.457720 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc"] Jul 22 19:35:56.459844 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:35:56.459819 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54de62b5_5f44_4de5_9b7d_af353adc84a0.slice/crio-ce9144616a145788b7059eea67d595ce0365d78e67798a101df9c64a7df6ef87 WatchSource:0}: Error finding container ce9144616a145788b7059eea67d595ce0365d78e67798a101df9c64a7df6ef87: Status 404 returned error can't find the container with id ce9144616a145788b7059eea67d595ce0365d78e67798a101df9c64a7df6ef87 Jul 22 19:35:56.627295 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:56.627201 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" event={"ID":"54de62b5-5f44-4de5-9b7d-af353adc84a0","Type":"ContainerStarted","Data":"ce9144616a145788b7059eea67d595ce0365d78e67798a101df9c64a7df6ef87"} Jul 22 19:35:58.642393 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:58.642356 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" event={"ID":"54de62b5-5f44-4de5-9b7d-af353adc84a0","Type":"ContainerStarted","Data":"7dfcb3b2cb0ec3f688fb2ef26a5d10930ddb6acbc9aeb683bfd9053fdbe164a3"} Jul 22 19:35:58.642935 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:58.642449 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" Jul 22 19:35:58.661413 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:35:58.661353 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" podStartSLOduration=2.041476873 podStartE2EDuration="3.661335858s" totalImagesPullingTime="1.619858985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:35:58.659107639 +0000 UTC m=+2361.571271722" watchObservedRunningTime="2026-07-22 19:35:58.661335858 +0000 UTC m=+2361.573499980" Jul 22 19:36:04.195019 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:04.194983 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr"] Jul 22 19:36:04.199183 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:04.199158 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" Jul 22 19:36:04.201860 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:04.201840 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-4addf8-serving-cert\"" Jul 22 19:36:04.212121 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:04.212094 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr"] Jul 22 19:36:04.256551 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:04.256517 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gslzv\" (UniqueName: \"kubernetes.io/projected/069d181e-e949-421b-a306-bc6b60ee08eb-kube-api-access-gslzv\") pod \"maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr\" (UID: \"069d181e-e949-421b-a306-bc6b60ee08eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" Jul 22 19:36:04.256747 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:04.256558 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069d181e-e949-421b-a306-bc6b60ee08eb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr\" (UID: \"069d181e-e949-421b-a306-bc6b60ee08eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" Jul 22 19:36:04.256747 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:04.256641 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069d181e-e949-421b-a306-bc6b60ee08eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr\" (UID: \"069d181e-e949-421b-a306-bc6b60ee08eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" Jul 22 19:36:04.357851 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:04.357808 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gslzv\" (UniqueName: \"kubernetes.io/projected/069d181e-e949-421b-a306-bc6b60ee08eb-kube-api-access-gslzv\") pod \"maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr\" (UID: \"069d181e-e949-421b-a306-bc6b60ee08eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" Jul 22 19:36:04.358066 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:04.357859 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069d181e-e949-421b-a306-bc6b60ee08eb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr\" (UID: \"069d181e-e949-421b-a306-bc6b60ee08eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" Jul 22 19:36:04.358066 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:04.357912 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069d181e-e949-421b-a306-bc6b60ee08eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr\" (UID: \"069d181e-e949-421b-a306-bc6b60ee08eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" Jul 22 19:36:04.358155 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:36:04.358108 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-serving-cert: secret "maas-api-e2e-shared-b-4addf8-serving-cert" not found Jul 22 19:36:04.358191 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:36:04.358169 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/069d181e-e949-421b-a306-bc6b60ee08eb-maas-api-tls podName:069d181e-e949-421b-a306-bc6b60ee08eb nodeName:}" failed. No retries permitted until 2026-07-22 19:36:04.858153193 +0000 UTC m=+2367.770317243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/069d181e-e949-421b-a306-bc6b60ee08eb-maas-api-tls") pod "maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" (UID: "069d181e-e949-421b-a306-bc6b60ee08eb") : secret "maas-api-e2e-shared-b-4addf8-serving-cert" not found Jul 22 19:36:04.358615 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:04.358597 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069d181e-e949-421b-a306-bc6b60ee08eb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr\" (UID: \"069d181e-e949-421b-a306-bc6b60ee08eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" Jul 22 19:36:04.371116 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:04.371080 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gslzv\" (UniqueName: \"kubernetes.io/projected/069d181e-e949-421b-a306-bc6b60ee08eb-kube-api-access-gslzv\") pod \"maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr\" (UID: \"069d181e-e949-421b-a306-bc6b60ee08eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" Jul 22 19:36:04.654237 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:04.654207 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" Jul 22 19:36:04.862866 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:04.862829 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069d181e-e949-421b-a306-bc6b60ee08eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr\" (UID: \"069d181e-e949-421b-a306-bc6b60ee08eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" Jul 22 19:36:04.865731 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:04.865697 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069d181e-e949-421b-a306-bc6b60ee08eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr\" (UID: \"069d181e-e949-421b-a306-bc6b60ee08eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" Jul 22 19:36:05.109931 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:05.109894 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" Jul 22 19:36:05.282412 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:05.282379 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr"] Jul 22 19:36:05.283569 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:36:05.283539 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069d181e_e949_421b_a306_bc6b60ee08eb.slice/crio-fcb4c1582e3bf4f3684d054655c9b7caba185bf1753ae4bdf079791f73f200a1 WatchSource:0}: Error finding container fcb4c1582e3bf4f3684d054655c9b7caba185bf1753ae4bdf079791f73f200a1: Status 404 returned error can't find the container with id fcb4c1582e3bf4f3684d054655c9b7caba185bf1753ae4bdf079791f73f200a1 Jul 22 19:36:05.687962 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:05.687917 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" event={"ID":"069d181e-e949-421b-a306-bc6b60ee08eb","Type":"ContainerStarted","Data":"fcb4c1582e3bf4f3684d054655c9b7caba185bf1753ae4bdf079791f73f200a1"} Jul 22 19:36:06.695993 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:06.695954 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" event={"ID":"069d181e-e949-421b-a306-bc6b60ee08eb","Type":"ContainerStarted","Data":"b594ee8ba0a36672663fd439a849f0339b85a2f81ae3a1249390c640dee19541"} Jul 22 19:36:06.696487 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:06.696030 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" Jul 22 19:36:06.712281 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:06.712225 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" podStartSLOduration=2.21523337 podStartE2EDuration="2.712207696s" totalImagesPullingTime="496.974326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:36:06.710886599 +0000 UTC m=+2369.623050665" watchObservedRunningTime="2026-07-22 19:36:06.712207696 +0000 UTC m=+2369.624371764" Jul 22 19:36:12.707908 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:12.707880 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" Jul 22 19:36:37.759090 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:37.759057 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:36:37.780219 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:37.780179 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:36:37.962432 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:37.962395 2583 scope.go:117] "RemoveContainer" containerID="5b705011ade99d86150fb1a1ef4102ae0a0a8f3318fc851b113c3a3a0c35dfe4" Jul 22 19:36:37.971574 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:36:37.971552 2583 scope.go:117] "RemoveContainer" containerID="6413956e86420277648bdf0f8e3460505032f67c38cfb49983312f10b9c87f58" Jul 22 19:40:38.035211 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:40:38.035179 2583 scope.go:117] "RemoveContainer" containerID="c60f4be015a2cdd9bc36c468de949dacc85dba0aa7ad07ab1d1cb77007f7fc59" Jul 22 19:40:38.045022 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:40:38.044997 2583 scope.go:117] "RemoveContainer" containerID="dd5ce1f766f5e9b423521b864a89a08f3236792778407091cfa4bf9646cc64b4" Jul 22 19:41:37.800421 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:41:37.800393 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:41:37.823100 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:41:37.823074 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:41:38.072418 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:41:38.072338 2583 scope.go:117] "RemoveContainer" containerID="26b8cb36773a2f1e8f30aaf3a016d66028c4c76792c99a92b1d1635047ec4925" Jul 22 19:41:38.081722 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:41:38.081699 2583 scope.go:117] "RemoveContainer" containerID="55b46fb1864067723c6f7c3f3662a36ecc07435fb7cb831cc5e58a38a05e68fc" Jul 22 19:42:36.503781 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:36.503745 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc"] Jul 22 19:42:36.504364 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:36.504326 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" podUID="54de62b5-5f44-4de5-9b7d-af353adc84a0" containerName="maas-api" containerID="cri-o://7dfcb3b2cb0ec3f688fb2ef26a5d10930ddb6acbc9aeb683bfd9053fdbe164a3" gracePeriod=30 Jul 22 19:42:36.768263 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:36.768238 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" Jul 22 19:42:36.903586 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:36.903551 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9rb9\" (UniqueName: \"kubernetes.io/projected/54de62b5-5f44-4de5-9b7d-af353adc84a0-kube-api-access-x9rb9\") pod \"54de62b5-5f44-4de5-9b7d-af353adc84a0\" (UID: \"54de62b5-5f44-4de5-9b7d-af353adc84a0\") " Jul 22 19:42:36.903767 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:36.903600 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54de62b5-5f44-4de5-9b7d-af353adc84a0-maas-api-tls\") pod \"54de62b5-5f44-4de5-9b7d-af353adc84a0\" (UID: \"54de62b5-5f44-4de5-9b7d-af353adc84a0\") " Jul 22 19:42:36.903767 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:36.903626 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54de62b5-5f44-4de5-9b7d-af353adc84a0-openshift-service-ca\") pod \"54de62b5-5f44-4de5-9b7d-af353adc84a0\" (UID: \"54de62b5-5f44-4de5-9b7d-af353adc84a0\") " Jul 22 19:42:36.904044 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:36.904020 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54de62b5-5f44-4de5-9b7d-af353adc84a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "54de62b5-5f44-4de5-9b7d-af353adc84a0" (UID: "54de62b5-5f44-4de5-9b7d-af353adc84a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:42:36.905799 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:36.905769 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54de62b5-5f44-4de5-9b7d-af353adc84a0-kube-api-access-x9rb9" (OuterVolumeSpecName: "kube-api-access-x9rb9") pod "54de62b5-5f44-4de5-9b7d-af353adc84a0" (UID: "54de62b5-5f44-4de5-9b7d-af353adc84a0"). InnerVolumeSpecName "kube-api-access-x9rb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:42:36.905907 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:36.905869 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54de62b5-5f44-4de5-9b7d-af353adc84a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "54de62b5-5f44-4de5-9b7d-af353adc84a0" (UID: "54de62b5-5f44-4de5-9b7d-af353adc84a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:42:37.005097 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:37.005059 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54de62b5-5f44-4de5-9b7d-af353adc84a0-openshift-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:42:37.005097 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:37.005091 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9rb9\" (UniqueName: \"kubernetes.io/projected/54de62b5-5f44-4de5-9b7d-af353adc84a0-kube-api-access-x9rb9\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:42:37.005097 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:37.005101 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54de62b5-5f44-4de5-9b7d-af353adc84a0-maas-api-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:42:37.111365 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:37.111279 2583 generic.go:356] "Generic (PLEG): container finished" podID="54de62b5-5f44-4de5-9b7d-af353adc84a0" containerID="7dfcb3b2cb0ec3f688fb2ef26a5d10930ddb6acbc9aeb683bfd9053fdbe164a3" exitCode=0 Jul 22 19:42:37.111365 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:37.111325 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" Jul 22 19:42:37.111365 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:37.111335 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" event={"ID":"54de62b5-5f44-4de5-9b7d-af353adc84a0","Type":"ContainerDied","Data":"7dfcb3b2cb0ec3f688fb2ef26a5d10930ddb6acbc9aeb683bfd9053fdbe164a3"} Jul 22 19:42:37.111613 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:37.111379 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc" event={"ID":"54de62b5-5f44-4de5-9b7d-af353adc84a0","Type":"ContainerDied","Data":"ce9144616a145788b7059eea67d595ce0365d78e67798a101df9c64a7df6ef87"} Jul 22 19:42:37.111613 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:37.111401 2583 scope.go:117] "RemoveContainer" containerID="7dfcb3b2cb0ec3f688fb2ef26a5d10930ddb6acbc9aeb683bfd9053fdbe164a3" Jul 22 19:42:37.121366 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:37.121345 2583 scope.go:117] "RemoveContainer" containerID="7dfcb3b2cb0ec3f688fb2ef26a5d10930ddb6acbc9aeb683bfd9053fdbe164a3" Jul 22 19:42:37.121672 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:42:37.121641 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dfcb3b2cb0ec3f688fb2ef26a5d10930ddb6acbc9aeb683bfd9053fdbe164a3\": container with ID starting with 7dfcb3b2cb0ec3f688fb2ef26a5d10930ddb6acbc9aeb683bfd9053fdbe164a3 not found: ID does not exist" containerID="7dfcb3b2cb0ec3f688fb2ef26a5d10930ddb6acbc9aeb683bfd9053fdbe164a3" Jul 22 19:42:37.121757 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:37.121682 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dfcb3b2cb0ec3f688fb2ef26a5d10930ddb6acbc9aeb683bfd9053fdbe164a3"} err="failed to get container status \"7dfcb3b2cb0ec3f688fb2ef26a5d10930ddb6acbc9aeb683bfd9053fdbe164a3\": rpc error: code = NotFound desc = could not find container \"7dfcb3b2cb0ec3f688fb2ef26a5d10930ddb6acbc9aeb683bfd9053fdbe164a3\": container with ID starting with 7dfcb3b2cb0ec3f688fb2ef26a5d10930ddb6acbc9aeb683bfd9053fdbe164a3 not found: ID does not exist" Jul 22 19:42:37.132921 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:37.132900 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc"] Jul 22 19:42:37.136570 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:37.136548 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-938311-c6b7f7475-zlkvc"] Jul 22 19:42:37.551466 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:37.551436 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54de62b5-5f44-4de5-9b7d-af353adc84a0" path="/var/lib/kubelet/pods/54de62b5-5f44-4de5-9b7d-af353adc84a0/volumes" Jul 22 19:42:48.048391 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:48.048354 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc"] Jul 22 19:42:48.048814 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:48.048802 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54de62b5-5f44-4de5-9b7d-af353adc84a0" containerName="maas-api" Jul 22 19:42:48.048814 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:48.048816 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="54de62b5-5f44-4de5-9b7d-af353adc84a0" containerName="maas-api" Jul 22 19:42:48.048922 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:48.048887 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="54de62b5-5f44-4de5-9b7d-af353adc84a0" containerName="maas-api" Jul 22 19:42:48.052093 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:48.052075 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc" Jul 22 19:42:48.054412 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:48.054393 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jhv4j\"" Jul 22 19:42:48.059851 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:48.059814 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc"] Jul 22 19:42:48.215904 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:48.215866 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c2c377-9bd5-431c-80a8-b7129a38ab86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc\" (UID: \"55c2c377-9bd5-431c-80a8-b7129a38ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc" Jul 22 19:42:48.316526 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:48.316415 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c2c377-9bd5-431c-80a8-b7129a38ab86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc\" (UID: \"55c2c377-9bd5-431c-80a8-b7129a38ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc" Jul 22 19:42:48.317040 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:48.317019 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c2c377-9bd5-431c-80a8-b7129a38ab86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc\" (UID: \"55c2c377-9bd5-431c-80a8-b7129a38ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc" Jul 22 19:42:48.364562 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:48.364529 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc" Jul 22 19:42:48.513027 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:48.513000 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc"] Jul 22 19:42:48.514349 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:42:48.514324 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c2c377_9bd5_431c_80a8_b7129a38ab86.slice/crio-f07d2d9cb5f4af152747326d2eb1284350e62b9039ea013b91416a9318ef09a9 WatchSource:0}: Error finding container f07d2d9cb5f4af152747326d2eb1284350e62b9039ea013b91416a9318ef09a9: Status 404 returned error can't find the container with id f07d2d9cb5f4af152747326d2eb1284350e62b9039ea013b91416a9318ef09a9 Jul 22 19:42:49.192682 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:49.192636 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc" event={"ID":"55c2c377-9bd5-431c-80a8-b7129a38ab86","Type":"ContainerStarted","Data":"f85c0dfa65fab00f160c054433a3110bba4cc82b960f81b5e80d8dfaf5c2d3f5"} Jul 22 19:42:49.193145 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:49.192747 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc" event={"ID":"55c2c377-9bd5-431c-80a8-b7129a38ab86","Type":"ContainerStarted","Data":"f07d2d9cb5f4af152747326d2eb1284350e62b9039ea013b91416a9318ef09a9"} Jul 22 19:42:49.209253 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:49.209185 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc" podStartSLOduration=1.2091675849999999 podStartE2EDuration="1.209167585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:42:49.208696657 +0000 UTC m=+2772.120860727" watchObservedRunningTime="2026-07-22 19:42:49.209167585 +0000 UTC m=+2772.121331656" Jul 22 19:42:50.200142 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:50.200108 2583 generic.go:356] "Generic (PLEG): container finished" podID="55c2c377-9bd5-431c-80a8-b7129a38ab86" containerID="f85c0dfa65fab00f160c054433a3110bba4cc82b960f81b5e80d8dfaf5c2d3f5" exitCode=0 Jul 22 19:42:50.200567 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:50.200194 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc" event={"ID":"55c2c377-9bd5-431c-80a8-b7129a38ab86","Type":"ContainerDied","Data":"f85c0dfa65fab00f160c054433a3110bba4cc82b960f81b5e80d8dfaf5c2d3f5"} Jul 22 19:42:51.339838 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:51.339810 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc" Jul 22 19:42:51.449456 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:51.449426 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c2c377-9bd5-431c-80a8-b7129a38ab86-maas-api-service-ca\") pod \"55c2c377-9bd5-431c-80a8-b7129a38ab86\" (UID: \"55c2c377-9bd5-431c-80a8-b7129a38ab86\") " Jul 22 19:42:51.449821 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:51.449798 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c2c377-9bd5-431c-80a8-b7129a38ab86-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55c2c377-9bd5-431c-80a8-b7129a38ab86" (UID: "55c2c377-9bd5-431c-80a8-b7129a38ab86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:42:51.550728 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:51.550700 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c2c377-9bd5-431c-80a8-b7129a38ab86-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:42:52.213620 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:52.213586 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc" Jul 22 19:42:52.213620 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:52.213616 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc" event={"ID":"55c2c377-9bd5-431c-80a8-b7129a38ab86","Type":"ContainerDied","Data":"f07d2d9cb5f4af152747326d2eb1284350e62b9039ea013b91416a9318ef09a9"} Jul 22 19:42:52.213832 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:52.213659 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f07d2d9cb5f4af152747326d2eb1284350e62b9039ea013b91416a9318ef09a9" Jul 22 19:42:58.426817 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:58.426577 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr"] Jul 22 19:42:58.427414 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:58.426827 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" podUID="069d181e-e949-421b-a306-bc6b60ee08eb" containerName="maas-api" containerID="cri-o://b594ee8ba0a36672663fd439a849f0339b85a2f81ae3a1249390c640dee19541" gracePeriod=30 Jul 22 19:42:58.695098 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:58.695074 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" Jul 22 19:42:58.824281 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:58.824247 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069d181e-e949-421b-a306-bc6b60ee08eb-maas-api-tls\") pod \"069d181e-e949-421b-a306-bc6b60ee08eb\" (UID: \"069d181e-e949-421b-a306-bc6b60ee08eb\") " Jul 22 19:42:58.824461 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:58.824325 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gslzv\" (UniqueName: \"kubernetes.io/projected/069d181e-e949-421b-a306-bc6b60ee08eb-kube-api-access-gslzv\") pod \"069d181e-e949-421b-a306-bc6b60ee08eb\" (UID: \"069d181e-e949-421b-a306-bc6b60ee08eb\") " Jul 22 19:42:58.824461 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:58.824377 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069d181e-e949-421b-a306-bc6b60ee08eb-openshift-service-ca\") pod \"069d181e-e949-421b-a306-bc6b60ee08eb\" (UID: \"069d181e-e949-421b-a306-bc6b60ee08eb\") " Jul 22 19:42:58.824787 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:58.824760 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/069d181e-e949-421b-a306-bc6b60ee08eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "069d181e-e949-421b-a306-bc6b60ee08eb" (UID: "069d181e-e949-421b-a306-bc6b60ee08eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 19:42:58.826555 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:58.826526 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/069d181e-e949-421b-a306-bc6b60ee08eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "069d181e-e949-421b-a306-bc6b60ee08eb" (UID: "069d181e-e949-421b-a306-bc6b60ee08eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 19:42:58.826649 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:58.826526 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069d181e-e949-421b-a306-bc6b60ee08eb-kube-api-access-gslzv" (OuterVolumeSpecName: "kube-api-access-gslzv") pod "069d181e-e949-421b-a306-bc6b60ee08eb" (UID: "069d181e-e949-421b-a306-bc6b60ee08eb"). InnerVolumeSpecName "kube-api-access-gslzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 19:42:58.925757 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:58.925721 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069d181e-e949-421b-a306-bc6b60ee08eb-maas-api-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:42:58.925757 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:58.925751 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gslzv\" (UniqueName: \"kubernetes.io/projected/069d181e-e949-421b-a306-bc6b60ee08eb-kube-api-access-gslzv\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:42:58.925757 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:58.925762 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069d181e-e949-421b-a306-bc6b60ee08eb-openshift-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Jul 22 19:42:59.256467 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:59.256434 2583 generic.go:356] "Generic (PLEG): container finished" podID="069d181e-e949-421b-a306-bc6b60ee08eb" containerID="b594ee8ba0a36672663fd439a849f0339b85a2f81ae3a1249390c640dee19541" exitCode=0 Jul 22 19:42:59.256693 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:59.256477 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" Jul 22 19:42:59.256693 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:59.256533 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" event={"ID":"069d181e-e949-421b-a306-bc6b60ee08eb","Type":"ContainerDied","Data":"b594ee8ba0a36672663fd439a849f0339b85a2f81ae3a1249390c640dee19541"} Jul 22 19:42:59.256693 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:59.256595 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr" event={"ID":"069d181e-e949-421b-a306-bc6b60ee08eb","Type":"ContainerDied","Data":"fcb4c1582e3bf4f3684d054655c9b7caba185bf1753ae4bdf079791f73f200a1"} Jul 22 19:42:59.256693 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:59.256626 2583 scope.go:117] "RemoveContainer" containerID="b594ee8ba0a36672663fd439a849f0339b85a2f81ae3a1249390c640dee19541" Jul 22 19:42:59.268137 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:59.267745 2583 scope.go:117] "RemoveContainer" containerID="b594ee8ba0a36672663fd439a849f0339b85a2f81ae3a1249390c640dee19541" Jul 22 19:42:59.268806 ip-10-0-139-51 kubenswrapper[2583]: E0722 19:42:59.268771 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b594ee8ba0a36672663fd439a849f0339b85a2f81ae3a1249390c640dee19541\": container with ID starting with b594ee8ba0a36672663fd439a849f0339b85a2f81ae3a1249390c640dee19541 not found: ID does not exist" containerID="b594ee8ba0a36672663fd439a849f0339b85a2f81ae3a1249390c640dee19541" Jul 22 19:42:59.268907 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:59.268814 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b594ee8ba0a36672663fd439a849f0339b85a2f81ae3a1249390c640dee19541"} err="failed to get container status \"b594ee8ba0a36672663fd439a849f0339b85a2f81ae3a1249390c640dee19541\": rpc error: code = NotFound desc = could not find container \"b594ee8ba0a36672663fd439a849f0339b85a2f81ae3a1249390c640dee19541\": container with ID starting with b594ee8ba0a36672663fd439a849f0339b85a2f81ae3a1249390c640dee19541 not found: ID does not exist" Jul 22 19:42:59.278829 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:59.278807 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr"] Jul 22 19:42:59.282808 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:59.282785 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4addf8-c6bbd494b-t6whr"] Jul 22 19:42:59.550843 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:42:59.550812 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="069d181e-e949-421b-a306-bc6b60ee08eb" path="/var/lib/kubelet/pods/069d181e-e949-421b-a306-bc6b60ee08eb/volumes" Jul 22 19:43:03.127053 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:03.127008 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc"] Jul 22 19:43:03.129542 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:03.129512 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4addf8-2zrlc"] Jul 22 19:43:03.551052 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:03.551014 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55c2c377-9bd5-431c-80a8-b7129a38ab86" path="/var/lib/kubelet/pods/55c2c377-9bd5-431c-80a8-b7129a38ab86/volumes" Jul 22 19:43:15.736785 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:15.736758 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5cc9h_9259a9ed-840b-4ed8-85bc-9dcb39cba2e2/manager/0.log" Jul 22 19:43:15.855995 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:15.855965 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4qf2d_57bf3d99-f6f5-4763-8ca0-b2c91840d095/manager/0.log" Jul 22 19:43:15.978289 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:15.978262 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d64d87758-2fzpf_d59b3764-c264-4cf1-8d6c-dadc1c47229f/manager/0.log" Jul 22 19:43:16.097889 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:16.097859 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-f77r6_37f04a4f-6db5-48f7-b64d-10002f1949f5/server/0.log" Jul 22 19:43:16.219354 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:16.219322 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-85kgq_90c5840d-8097-4156-b044-f9b7e3e4e423/manager/0.log" Jul 22 19:43:16.338904 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:16.338874 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cb586fcb5-k8xhk_fea8ac18-f5da-4d52-b696-c912e93c370a/copy-manifests/0.log" Jul 22 19:43:16.345767 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:16.345742 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cb586fcb5-k8xhk_fea8ac18-f5da-4d52-b696-c912e93c370a/manager/0.log" Jul 22 19:43:16.933656 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:16.933626 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-57bfdff96-9487z_24004b0e-bb1c-4af6-aec6-86b7b16e9621/maas-api/0.log" Jul 22 19:43:17.046009 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:17.045979 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29745795-d5x4x_80b20b3c-fb49-4543-a19e-edbf415574d3/cleanup/0.log" Jul 22 19:43:17.154590 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:17.154555 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29745810-75xxn_b69fce0b-0962-4daf-aa4e-189d9361903d/cleanup/0.log" Jul 22 19:43:17.263940 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:17.263858 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-8mb8z_5ef60f74-9c3b-4e44-8e2d-fbd91d5a06cb/postgres/0.log" Jul 22 19:43:18.026919 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:18.026884 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd_6371b6ea-2108-4ab2-8d67-8ded5127613a/extract/0.log" Jul 22 19:43:18.032553 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:18.032533 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd_6371b6ea-2108-4ab2-8d67-8ded5127613a/util/0.log" Jul 22 19:43:18.038641 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:18.038617 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd_6371b6ea-2108-4ab2-8d67-8ded5127613a/pull/0.log" Jul 22 19:43:18.152257 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:18.152229 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq_334f8642-4498-4c8c-9c13-d810152519fa/util/0.log" Jul 22 19:43:18.158577 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:18.158555 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq_334f8642-4498-4c8c-9c13-d810152519fa/pull/0.log" Jul 22 19:43:18.165159 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:18.165139 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq_334f8642-4498-4c8c-9c13-d810152519fa/extract/0.log" Jul 22 19:43:18.280737 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:18.280662 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6_fc860783-da97-4027-bce5-75d91c7dd87d/util/0.log" Jul 22 19:43:18.287655 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:18.287632 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6_fc860783-da97-4027-bce5-75d91c7dd87d/pull/0.log" Jul 22 19:43:18.294902 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:18.294878 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6_fc860783-da97-4027-bce5-75d91c7dd87d/extract/0.log" Jul 22 19:43:18.398024 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:18.397998 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn_fd614366-354d-4198-a00b-c4bad2d27b92/util/0.log" Jul 22 19:43:18.403706 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:18.403681 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn_fd614366-354d-4198-a00b-c4bad2d27b92/pull/0.log" Jul 22 19:43:18.409302 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:18.409282 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn_fd614366-354d-4198-a00b-c4bad2d27b92/extract/0.log" Jul 22 19:43:19.329643 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:19.329615 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hkkt6_4d1f34a2-bf52-4511-9596-d1aaef7f0031/manager/0.log" Jul 22 19:43:19.788268 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:19.788234 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-kc42t_b6eb0a8d-45b6-4f82-af15-5e714fac5c52/discovery/0.log" Jul 22 19:43:19.900645 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:19.900611 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69768f7b48-hm7nn_1639fe19-20ec-44cd-b23b-167138c4e510/kube-auth-proxy/0.log" Jul 22 19:43:20.244196 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:20.244112 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-rr9rg_31cb3166-0d51-40f5-98ad-aa881f0b6ffb/payload-pre-processing/0.log" Jul 22 19:43:20.486440 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:20.486415 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8494c9898d-4jsgr_dd794227-b4ac-45cb-ab35-ba8f431d0948/router/0.log" Jul 22 19:43:20.825177 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:20.825140 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-zvngw_9805fea0-87e3-40fd-9534-feea14fb01a1/main/0.log" Jul 22 19:43:20.933117 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:20.933090 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-4s49m_b59b394f-42c8-4a7e-a3e1-4cb9d378721d/main/0.log" Jul 22 19:43:29.257151 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:29.257123 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j79dm_5ac11ffa-90fb-40e6-a021-0984d4d8c9bd/global-pull-secret-syncer/0.log" Jul 22 19:43:29.376408 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:29.376377 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v2tfs_f9c816fc-7154-4b8d-86e9-85ef121f423d/konnectivity-agent/0.log" Jul 22 19:43:29.436509 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:29.436474 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-51.ec2.internal_f73e47efca17a7bdd186cdd86897ddfa/haproxy/0.log" Jul 22 19:43:33.949247 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:33.949155 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd_6371b6ea-2108-4ab2-8d67-8ded5127613a/extract/0.log" Jul 22 19:43:33.976529 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:33.976485 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd_6371b6ea-2108-4ab2-8d67-8ded5127613a/util/0.log" Jul 22 19:43:33.997424 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:33.997393 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592rtmd_6371b6ea-2108-4ab2-8d67-8ded5127613a/pull/0.log" Jul 22 19:43:34.032572 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:34.032541 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq_334f8642-4498-4c8c-9c13-d810152519fa/extract/0.log" Jul 22 19:43:34.061946 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:34.061915 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq_334f8642-4498-4c8c-9c13-d810152519fa/util/0.log" Jul 22 19:43:34.079537 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:34.079506 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vrfqq_334f8642-4498-4c8c-9c13-d810152519fa/pull/0.log" Jul 22 19:43:34.104853 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:34.104804 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6_fc860783-da97-4027-bce5-75d91c7dd87d/extract/0.log" Jul 22 19:43:34.124891 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:34.124865 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6_fc860783-da97-4027-bce5-75d91c7dd87d/util/0.log" Jul 22 19:43:34.145194 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:34.145163 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qtvw6_fc860783-da97-4027-bce5-75d91c7dd87d/pull/0.log" Jul 22 19:43:34.176866 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:34.176841 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn_fd614366-354d-4198-a00b-c4bad2d27b92/extract/0.log" Jul 22 19:43:34.195844 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:34.195818 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn_fd614366-354d-4198-a00b-c4bad2d27b92/util/0.log" Jul 22 19:43:34.214778 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:34.214710 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xshqn_fd614366-354d-4198-a00b-c4bad2d27b92/pull/0.log" Jul 22 19:43:35.155591 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:35.155559 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hkkt6_4d1f34a2-bf52-4511-9596-d1aaef7f0031/manager/0.log" Jul 22 19:43:36.870562 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:36.870530 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_afc0226a-ddaf-4667-b107-a027ee2b8d89/alertmanager/0.log" Jul 22 19:43:36.891968 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:36.891943 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_afc0226a-ddaf-4667-b107-a027ee2b8d89/config-reloader/0.log" Jul 22 19:43:36.914007 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:36.913982 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_afc0226a-ddaf-4667-b107-a027ee2b8d89/kube-rbac-proxy-web/0.log" Jul 22 19:43:36.934831 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:36.934804 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_afc0226a-ddaf-4667-b107-a027ee2b8d89/kube-rbac-proxy/0.log" Jul 22 19:43:36.957357 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:36.957288 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_afc0226a-ddaf-4667-b107-a027ee2b8d89/kube-rbac-proxy-metric/0.log" Jul 22 19:43:36.976350 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:36.976326 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_afc0226a-ddaf-4667-b107-a027ee2b8d89/prom-label-proxy/0.log" Jul 22 19:43:36.996660 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:36.996634 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_afc0226a-ddaf-4667-b107-a027ee2b8d89/init-config-reloader/0.log" Jul 22 19:43:37.035650 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:37.035618 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9rgk7_d5899683-5c18-4ee4-9878-2fec416d25a8/cluster-monitoring-operator/0.log" Jul 22 19:43:37.057170 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:37.057140 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-279fz_0ec28774-2cd9-4851-8907-6e8bd22a7b3b/kube-state-metrics/0.log" Jul 22 19:43:37.082894 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:37.082866 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-279fz_0ec28774-2cd9-4851-8907-6e8bd22a7b3b/kube-rbac-proxy-main/0.log" Jul 22 19:43:37.101283 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:37.101253 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-279fz_0ec28774-2cd9-4851-8907-6e8bd22a7b3b/kube-rbac-proxy-self/0.log" Jul 22 19:43:37.152011 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:37.151985 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-2cmfw_8782b139-b8d9-4ef9-877a-d68684d267f2/monitoring-plugin/0.log" Jul 22 19:43:37.332007 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:37.331979 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p2d9d_171ffb8e-7b10-4e0a-bef2-299ecff632ef/node-exporter/0.log" Jul 22 19:43:37.351472 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:37.351449 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p2d9d_171ffb8e-7b10-4e0a-bef2-299ecff632ef/kube-rbac-proxy/0.log" Jul 22 19:43:37.370067 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:37.370042 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p2d9d_171ffb8e-7b10-4e0a-bef2-299ecff632ef/init-textfile/0.log" Jul 22 19:43:37.399693 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:37.399667 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-lc8nf_8be1063f-28df-4a01-9546-23b7de7547ff/kube-rbac-proxy-main/0.log" Jul 22 19:43:37.418913 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:37.418891 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-lc8nf_8be1063f-28df-4a01-9546-23b7de7547ff/kube-rbac-proxy-self/0.log" Jul 22 19:43:37.438725 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:37.438700 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-lc8nf_8be1063f-28df-4a01-9546-23b7de7547ff/openshift-state-metrics/0.log" Jul 22 19:43:37.731280 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:37.731208 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59f988c449-c5x4p_59b62c85-911a-4519-ae05-e0f9a2d71907/telemeter-client/0.log" Jul 22 19:43:37.749687 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:37.749637 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59f988c449-c5x4p_59b62c85-911a-4519-ae05-e0f9a2d71907/reload/0.log" Jul 22 19:43:37.767816 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:37.767785 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59f988c449-c5x4p_59b62c85-911a-4519-ae05-e0f9a2d71907/kube-rbac-proxy/0.log" Jul 22 19:43:39.235157 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.235121 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45"] Jul 22 19:43:39.235580 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.235566 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="069d181e-e949-421b-a306-bc6b60ee08eb" containerName="maas-api" Jul 22 19:43:39.235629 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.235581 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="069d181e-e949-421b-a306-bc6b60ee08eb" containerName="maas-api" Jul 22 19:43:39.235629 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.235588 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c2c377-9bd5-431c-80a8-b7129a38ab86" containerName="revoke-keys" Jul 22 19:43:39.235629 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.235594 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c2c377-9bd5-431c-80a8-b7129a38ab86" containerName="revoke-keys" Jul 22 19:43:39.235727 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.235673 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c2c377-9bd5-431c-80a8-b7129a38ab86" containerName="revoke-keys" Jul 22 19:43:39.235727 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.235682 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="069d181e-e949-421b-a306-bc6b60ee08eb" containerName="maas-api" Jul 22 19:43:39.240136 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.240118 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.242695 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.242671 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mw4wd\"/\"openshift-service-ca.crt\"" Jul 22 19:43:39.242949 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.242930 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mw4wd\"/\"kube-root-ca.crt\"" Jul 22 19:43:39.243782 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.243763 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mw4wd\"/\"default-dockercfg-lwfnl\"" Jul 22 19:43:39.249044 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.249013 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45"] Jul 22 19:43:39.397767 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.397730 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17ad6c52-2b3c-4292-b513-16b145a35df2-sys\") pod \"perf-node-gather-daemonset-2qn45\" (UID: \"17ad6c52-2b3c-4292-b513-16b145a35df2\") " pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.397976 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.397794 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17ad6c52-2b3c-4292-b513-16b145a35df2-proc\") pod \"perf-node-gather-daemonset-2qn45\" (UID: \"17ad6c52-2b3c-4292-b513-16b145a35df2\") " pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.397976 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.397814 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17ad6c52-2b3c-4292-b513-16b145a35df2-lib-modules\") pod \"perf-node-gather-daemonset-2qn45\" (UID: \"17ad6c52-2b3c-4292-b513-16b145a35df2\") " pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.397976 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.397892 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17ad6c52-2b3c-4292-b513-16b145a35df2-podres\") pod \"perf-node-gather-daemonset-2qn45\" (UID: \"17ad6c52-2b3c-4292-b513-16b145a35df2\") " pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.397976 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.397935 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czjpp\" (UniqueName: \"kubernetes.io/projected/17ad6c52-2b3c-4292-b513-16b145a35df2-kube-api-access-czjpp\") pod \"perf-node-gather-daemonset-2qn45\" (UID: \"17ad6c52-2b3c-4292-b513-16b145a35df2\") " pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.499532 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.499419 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17ad6c52-2b3c-4292-b513-16b145a35df2-proc\") pod \"perf-node-gather-daemonset-2qn45\" (UID: \"17ad6c52-2b3c-4292-b513-16b145a35df2\") " pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.499532 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.499453 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17ad6c52-2b3c-4292-b513-16b145a35df2-lib-modules\") pod \"perf-node-gather-daemonset-2qn45\" (UID: \"17ad6c52-2b3c-4292-b513-16b145a35df2\") " pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.499532 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.499484 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17ad6c52-2b3c-4292-b513-16b145a35df2-podres\") pod \"perf-node-gather-daemonset-2qn45\" (UID: \"17ad6c52-2b3c-4292-b513-16b145a35df2\") " pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.499828 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.499563 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17ad6c52-2b3c-4292-b513-16b145a35df2-proc\") pod \"perf-node-gather-daemonset-2qn45\" (UID: \"17ad6c52-2b3c-4292-b513-16b145a35df2\") " pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.499828 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.499611 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17ad6c52-2b3c-4292-b513-16b145a35df2-podres\") pod \"perf-node-gather-daemonset-2qn45\" (UID: \"17ad6c52-2b3c-4292-b513-16b145a35df2\") " pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.499828 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.499615 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17ad6c52-2b3c-4292-b513-16b145a35df2-lib-modules\") pod \"perf-node-gather-daemonset-2qn45\" (UID: \"17ad6c52-2b3c-4292-b513-16b145a35df2\") " pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.499828 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.499628 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czjpp\" (UniqueName: \"kubernetes.io/projected/17ad6c52-2b3c-4292-b513-16b145a35df2-kube-api-access-czjpp\") pod \"perf-node-gather-daemonset-2qn45\" (UID: \"17ad6c52-2b3c-4292-b513-16b145a35df2\") " pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.499828 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.499762 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17ad6c52-2b3c-4292-b513-16b145a35df2-sys\") pod \"perf-node-gather-daemonset-2qn45\" (UID: \"17ad6c52-2b3c-4292-b513-16b145a35df2\") " pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.500018 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.499849 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17ad6c52-2b3c-4292-b513-16b145a35df2-sys\") pod \"perf-node-gather-daemonset-2qn45\" (UID: \"17ad6c52-2b3c-4292-b513-16b145a35df2\") " pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.507480 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.507450 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czjpp\" (UniqueName: \"kubernetes.io/projected/17ad6c52-2b3c-4292-b513-16b145a35df2-kube-api-access-czjpp\") pod \"perf-node-gather-daemonset-2qn45\" (UID: \"17ad6c52-2b3c-4292-b513-16b145a35df2\") " pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.551217 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.551190 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:39.652076 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.652045 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/1.log" Jul 22 19:43:39.658019 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.657994 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-scnm7_255f4f29-f31e-4d05-8544-d16d69b75e34/console-operator/2.log" Jul 22 19:43:39.692107 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:39.692076 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45"] Jul 22 19:43:39.692285 ip-10-0-139-51 kubenswrapper[2583]: W0722 19:43:39.692253 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod17ad6c52_2b3c_4292_b513_16b145a35df2.slice/crio-134897b04e4ac88c3b43aba641b9e722f61f33509fa6ba38df43641a2546e776 WatchSource:0}: Error finding container 134897b04e4ac88c3b43aba641b9e722f61f33509fa6ba38df43641a2546e776: Status 404 returned error can't find the container with id 134897b04e4ac88c3b43aba641b9e722f61f33509fa6ba38df43641a2546e776 Jul 22 19:43:40.232678 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:40.232597 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-598df6b7d6-zg2jr_18389c84-6ee5-46c8-a5aa-a6caac93d353/console/0.log" Jul 22 19:43:40.499202 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:40.499096 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" event={"ID":"17ad6c52-2b3c-4292-b513-16b145a35df2","Type":"ContainerStarted","Data":"ee963302cdc47d9d412cf873821a39c6797f077489472decc9b0b8a8efc54505"} Jul 22 19:43:40.499202 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:40.499164 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:40.499202 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:40.499186 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" event={"ID":"17ad6c52-2b3c-4292-b513-16b145a35df2","Type":"ContainerStarted","Data":"134897b04e4ac88c3b43aba641b9e722f61f33509fa6ba38df43641a2546e776"} Jul 22 19:43:40.514257 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:40.514198 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" podStartSLOduration=1.514180351 podStartE2EDuration="1.514180351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 19:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:43:40.513301292 +0000 UTC m=+2823.425465360" watchObservedRunningTime="2026-07-22 19:43:40.514180351 +0000 UTC m=+2823.426344418" Jul 22 19:43:40.880699 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:40.880674 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-2fmdc_ed584a0f-c300-43ea-9cb8-dc928e29f0e8/volume-data-source-validator/0.log" Jul 22 19:43:41.891785 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:41.891752 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dk7bn_d9d7ca2a-6211-4cf0-af77-49663ce893c3/dns/0.log" Jul 22 19:43:41.909331 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:41.909303 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dk7bn_d9d7ca2a-6211-4cf0-af77-49663ce893c3/kube-rbac-proxy/0.log" Jul 22 19:43:41.973107 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:41.973073 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2fd99_0961d651-b746-4c2e-85f5-d2a00c044e3a/dns-node-resolver/0.log" Jul 22 19:43:42.577455 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:42.577423 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fxb82_00c3d60d-32a0-41ce-9ce6-45b51a3dd7da/node-ca/0.log" Jul 22 19:43:43.642557 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:43.642524 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-kc42t_b6eb0a8d-45b6-4f82-af15-5e714fac5c52/discovery/0.log" Jul 22 19:43:43.661987 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:43.661955 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69768f7b48-hm7nn_1639fe19-20ec-44cd-b23b-167138c4e510/kube-auth-proxy/0.log" Jul 22 19:43:43.771087 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:43.771057 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-rr9rg_31cb3166-0d51-40f5-98ad-aa881f0b6ffb/payload-pre-processing/0.log" Jul 22 19:43:43.828316 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:43.828284 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8494c9898d-4jsgr_dd794227-b4ac-45cb-ab35-ba8f431d0948/router/0.log" Jul 22 19:43:44.527789 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:44.527757 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zmgjp_2092fe4a-f842-4ebf-876e-3fe8b8ebb0a4/serve-healthcheck-canary/0.log" Jul 22 19:43:44.971326 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:44.971240 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-2nd4q_e0aa0a66-a4d6-4557-bf05-05c34c80c173/insights-operator/0.log" Jul 22 19:43:44.973142 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:44.973121 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-2nd4q_e0aa0a66-a4d6-4557-bf05-05c34c80c173/insights-operator/1.log" Jul 22 19:43:45.054773 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:45.054736 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bxp4d_300c2416-b0a7-4abc-93c9-da7234dd1d2e/kube-rbac-proxy/0.log" Jul 22 19:43:45.073375 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:45.073348 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bxp4d_300c2416-b0a7-4abc-93c9-da7234dd1d2e/exporter/0.log" Jul 22 19:43:45.093172 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:45.093147 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bxp4d_300c2416-b0a7-4abc-93c9-da7234dd1d2e/extractor/0.log" Jul 22 19:43:46.515512 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:46.515465 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mw4wd/perf-node-gather-daemonset-2qn45" Jul 22 19:43:47.189012 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:47.188983 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5cc9h_9259a9ed-840b-4ed8-85bc-9dcb39cba2e2/manager/0.log" Jul 22 19:43:47.373738 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:47.373704 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4qf2d_57bf3d99-f6f5-4763-8ca0-b2c91840d095/manager/0.log" Jul 22 19:43:47.486872 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:47.486774 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d64d87758-2fzpf_d59b3764-c264-4cf1-8d6c-dadc1c47229f/manager/0.log" Jul 22 19:43:47.505911 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:47.505880 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-f77r6_37f04a4f-6db5-48f7-b64d-10002f1949f5/server/0.log" Jul 22 19:43:47.572402 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:47.572370 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-85kgq_90c5840d-8097-4156-b044-f9b7e3e4e423/manager/0.log" Jul 22 19:43:47.594959 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:47.594927 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cb586fcb5-k8xhk_fea8ac18-f5da-4d52-b696-c912e93c370a/manager/0.log" Jul 22 19:43:47.614051 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:47.614028 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cb586fcb5-k8xhk_fea8ac18-f5da-4d52-b696-c912e93c370a/copy-manifests/0.log" Jul 22 19:43:49.252263 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:49.252231 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-6pcxq_b8cd6344-3e70-4fa9-8f70-49e5e8dc1ac2/openshift-lws-operator/0.log" Jul 22 19:43:50.475860 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:50.475824 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-kc42t_b6eb0a8d-45b6-4f82-af15-5e714fac5c52/discovery/0.log" Jul 22 19:43:50.488225 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:50.488201 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69768f7b48-hm7nn_1639fe19-20ec-44cd-b23b-167138c4e510/kube-auth-proxy/0.log" Jul 22 19:43:50.585253 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:50.585223 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-rr9rg_31cb3166-0d51-40f5-98ad-aa881f0b6ffb/payload-pre-processing/0.log" Jul 22 19:43:50.636741 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:50.636711 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8494c9898d-4jsgr_dd794227-b4ac-45cb-ab35-ba8f431d0948/router/0.log" Jul 22 19:43:51.906248 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:51.906208 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-kc42t_b6eb0a8d-45b6-4f82-af15-5e714fac5c52/discovery/0.log" Jul 22 19:43:51.917700 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:51.917675 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69768f7b48-hm7nn_1639fe19-20ec-44cd-b23b-167138c4e510/kube-auth-proxy/0.log" Jul 22 19:43:52.013510 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:52.013464 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-rr9rg_31cb3166-0d51-40f5-98ad-aa881f0b6ffb/payload-pre-processing/0.log" Jul 22 19:43:52.062292 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:52.062259 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8494c9898d-4jsgr_dd794227-b4ac-45cb-ab35-ba8f431d0948/router/0.log" Jul 22 19:43:54.518485 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:54.518456 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-jhcsn_aa91b8fa-def9-4745-b1ec-66df9c2adc5a/kube-storage-version-migrator-operator/1.log" Jul 22 19:43:54.519297 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:54.519280 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-jhcsn_aa91b8fa-def9-4745-b1ec-66df9c2adc5a/kube-storage-version-migrator-operator/0.log" Jul 22 19:43:55.573123 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:55.573094 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t6qvj_85d1fd76-3e08-49cc-8b59-81d1a2c69867/kube-multus-additional-cni-plugins/0.log" Jul 22 19:43:55.592466 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:55.592442 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t6qvj_85d1fd76-3e08-49cc-8b59-81d1a2c69867/egress-router-binary-copy/0.log" Jul 22 19:43:55.610897 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:55.610867 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t6qvj_85d1fd76-3e08-49cc-8b59-81d1a2c69867/cni-plugins/0.log" Jul 22 19:43:55.630425 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:55.630401 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t6qvj_85d1fd76-3e08-49cc-8b59-81d1a2c69867/bond-cni-plugin/0.log" Jul 22 19:43:55.650671 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:55.650645 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t6qvj_85d1fd76-3e08-49cc-8b59-81d1a2c69867/routeoverride-cni/0.log" Jul 22 19:43:55.669856 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:55.669829 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t6qvj_85d1fd76-3e08-49cc-8b59-81d1a2c69867/whereabouts-cni-bincopy/0.log" Jul 22 19:43:55.688662 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:55.688636 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t6qvj_85d1fd76-3e08-49cc-8b59-81d1a2c69867/whereabouts-cni/0.log" Jul 22 19:43:55.899134 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:55.899049 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qtb5p_bb72b4c7-6047-4aa2-9604-58e177e0e98d/kube-multus/0.log" Jul 22 19:43:55.953087 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:55.953056 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p27ms_abb50681-d581-483f-a16a-bccdb437e24e/network-metrics-daemon/0.log" Jul 22 19:43:55.968436 ip-10-0-139-51 kubenswrapper[2583]: I0722 19:43:55.968403 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p27ms_abb50681-d581-483f-a16a-bccdb437e24e/kube-rbac-proxy/0.log"