Jul 16 12:44:50.597778 ip-10-0-138-198 systemd[1]: Starting Kubernetes Kubelet... Jul 16 12:44:51.052004 ip-10-0-138-198 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 12:44:51.052004 ip-10-0-138-198 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 12:44:51.052004 ip-10-0-138-198 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 12:44:51.052004 ip-10-0-138-198 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 12:44:51.052004 ip-10-0-138-198 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 12:44:51.054603 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.054474 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 12:44:51.060965 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.060945 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:44:51.060965 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.060961 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:44:51.060965 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.060966 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:44:51.060965 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.060970 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.060975 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.060979 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.060982 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.060986 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.060990 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.060993 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.060999 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061003 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061007 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061010 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061014 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061018 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061023 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061028 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061033 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061037 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061041 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061045 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:44:51.061186 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061049 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061055 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061061 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061065 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061070 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061074 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061078 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061082 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061086 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061090 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061095 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061098 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061103 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061107 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061111 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061116 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061123 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061129 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061134 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:44:51.061953 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061138 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061143 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061147 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061151 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061167 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061172 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061177 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061181 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061185 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061189 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061193 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061198 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061204 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061208 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061212 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061216 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061220 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061230 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061234 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061238 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:44:51.062400 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061242 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061249 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061255 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061260 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061265 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061269 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061274 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061279 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061285 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061289 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061293 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061297 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061302 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061306 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061310 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061315 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061319 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061324 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061329 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:44:51.063108 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061335 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061339 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061343 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061347 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061352 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061356 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061360 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061364 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061368 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061372 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061377 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061382 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061386 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061390 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061394 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061398 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061402 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061406 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.061410 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063415 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:44:51.063924 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063427 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063433 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063438 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063442 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063448 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063453 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063457 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063462 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063467 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063471 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063475 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063481 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063485 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063490 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063495 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063501 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063505 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063510 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063514 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063540 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:44:51.064745 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063545 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063549 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063553 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063557 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063561 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063565 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063569 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063573 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063577 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063582 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063586 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063594 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063598 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063602 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063607 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063611 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063615 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063619 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063624 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063628 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:44:51.065626 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063632 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063636 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063641 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063645 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063650 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063654 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063658 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063662 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063668 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063675 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063679 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063683 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063687 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063691 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063696 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063700 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063704 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063708 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063713 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:44:51.066478 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063717 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063721 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063725 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063729 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063736 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063740 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063744 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063748 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063752 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063757 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063763 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063769 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063774 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063778 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063782 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063787 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063791 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063794 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063799 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:44:51.066975 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063806 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063810 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063815 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063821 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063826 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063831 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063835 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063839 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063843 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063848 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063852 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063856 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063860 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063864 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063868 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063872 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063876 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063880 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063884 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:44:51.067544 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.063888 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064456 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064464 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064468 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064473 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064477 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064481 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064486 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064490 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064495 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064499 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064503 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064508 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064512 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064536 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064540 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064544 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064548 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064552 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064556 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:44:51.068067 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064560 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064565 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064569 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064573 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064578 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064582 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064586 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064589 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064593 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064598 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064602 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064606 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064611 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064615 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064620 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064623 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064627 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064632 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064636 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064640 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:44:51.068824 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064644 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064648 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064653 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064657 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064661 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064667 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064671 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064675 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064680 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064684 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064688 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064692 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064696 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064700 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064704 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064708 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064712 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064715 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064719 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064724 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:44:51.069311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064728 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064735 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064741 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064748 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064753 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064757 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064763 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064768 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064772 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064777 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064781 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064786 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064790 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064795 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064799 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064803 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064808 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064812 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064817 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:44:51.069897 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064822 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064826 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064831 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064835 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064839 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064842 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064846 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064851 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064855 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064859 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064863 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064867 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064871 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064878 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064883 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064888 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064892 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064896 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064901 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:44:51.070399 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064905 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.064909 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065013 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065030 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065040 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065047 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065053 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065058 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065064 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065071 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065076 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065081 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065087 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065092 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065097 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065102 2565 flags.go:64] FLAG: --cgroup-root="" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065107 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065112 2565 flags.go:64] FLAG: --client-ca-file="" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065116 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065121 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065127 2565 flags.go:64] FLAG: --cluster-domain="" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065131 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065136 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065141 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 12:44:51.070884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065147 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065153 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065158 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065163 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065168 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065173 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065177 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065182 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065187 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065191 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065201 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065206 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065211 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065216 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065220 2565 flags.go:64] FLAG: --enable-server="true" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065225 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065233 2565 flags.go:64] FLAG: --event-burst="100" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065238 2565 flags.go:64] FLAG: --event-qps="50" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065245 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065250 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065255 2565 flags.go:64] FLAG: --eviction-hard="" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065260 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065266 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065271 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065275 2565 flags.go:64] FLAG: --eviction-soft="" Jul 16 12:44:51.071601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065280 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065284 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065289 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065294 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065298 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065303 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065307 2565 flags.go:64] FLAG: --feature-gates="" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065313 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065318 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065323 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065328 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065333 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065338 2565 flags.go:64] FLAG: --help="false" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065343 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065347 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065352 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065357 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065362 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065367 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065372 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065376 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065380 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065385 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 12:44:51.072222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065389 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065394 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065400 2565 flags.go:64] FLAG: --kube-reserved="" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065405 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065409 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065414 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065418 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065424 2565 flags.go:64] FLAG: --lock-file="" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065428 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065432 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065437 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065446 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065450 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065455 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065459 2565 flags.go:64] FLAG: --logging-format="text" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065464 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065469 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065477 2565 flags.go:64] FLAG: --manifest-url="" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065481 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065488 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065493 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065499 2565 flags.go:64] FLAG: --max-pods="110" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065504 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065508 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065513 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 12:44:51.072881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065534 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065540 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065544 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065549 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065559 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065564 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065569 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065574 2565 flags.go:64] FLAG: --pod-cidr="" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065578 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065589 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065594 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065599 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065603 2565 flags.go:64] FLAG: --port="10250" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065608 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065613 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a964c45b362a8d2c" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065618 2565 flags.go:64] FLAG: --qos-reserved="" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065623 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065628 2565 flags.go:64] FLAG: --register-node="true" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065632 2565 flags.go:64] FLAG: --register-with-taints="" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065638 2565 flags.go:64] FLAG: --registry-burst="10" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065642 2565 flags.go:64] FLAG: --registry-qps="5" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065646 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065650 2565 flags.go:64] FLAG: --reserved-memory="" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065656 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065661 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 12:44:51.073482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065668 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065673 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065677 2565 flags.go:64] FLAG: --runonce="false" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065682 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065687 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065691 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065696 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065700 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065705 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065710 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065715 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065719 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065724 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065728 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065733 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065737 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065744 2565 flags.go:64] FLAG: --system-cgroups="" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065749 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065757 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065762 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065767 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065773 2565 flags.go:64] FLAG: --tls-min-version="" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065777 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065783 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065788 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 12:44:51.074457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065793 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065797 2565 flags.go:64] FLAG: --v="2" Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065803 2565 flags.go:64] FLAG: --version="false" Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065809 2565 flags.go:64] FLAG: --vmodule="" Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065815 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.065820 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.065999 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066008 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066012 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066017 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066021 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066025 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066029 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066033 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066038 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066044 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066050 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066055 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066060 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066064 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:44:51.075245 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066068 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066073 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066077 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066083 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066087 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066092 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066096 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066111 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066116 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066120 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066124 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066130 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066134 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066138 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066142 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066146 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066151 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066155 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066159 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066165 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:44:51.075809 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066170 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066174 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066178 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066182 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066186 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066190 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066194 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066198 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066202 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066207 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066211 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066215 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066219 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066222 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066226 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066232 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066236 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066241 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066245 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066249 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:44:51.076323 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066253 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066257 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066261 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066265 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066270 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066274 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066278 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066282 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066286 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066290 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066294 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066302 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066307 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066312 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066317 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066321 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066326 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066330 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066335 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066339 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:44:51.076823 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066343 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066347 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066351 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066354 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066359 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066363 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066367 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066372 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066376 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066380 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066384 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066388 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066394 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066399 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066404 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066408 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066412 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066417 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066421 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066425 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:44:51.077307 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066429 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:44:51.077806 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066433 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:44:51.077806 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066437 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:44:51.077806 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066443 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:44:51.077806 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.066447 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:44:51.077806 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.067274 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 12:44:51.077806 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.075969 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 12:44:51.077806 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.075984 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 12:44:51.077806 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.076003 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 12:44:51.077806 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.076012 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 12:44:51.077806 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076067 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:44:51.077806 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076073 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:44:51.077806 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076077 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:44:51.077806 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076080 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:44:51.077806 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076083 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076086 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076089 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076092 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076095 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076098 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076101 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076103 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076106 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076108 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076111 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076113 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076129 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076133 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076136 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076139 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076142 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076145 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076148 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076151 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:44:51.078147 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076154 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076156 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076159 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076161 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076163 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076166 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076169 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076171 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076173 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076176 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076178 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076182 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076186 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076189 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076192 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076194 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076197 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076200 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076202 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076205 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:44:51.078637 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076207 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076210 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076212 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076215 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076217 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076220 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076222 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076225 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076228 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076230 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076233 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076235 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076238 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076240 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076243 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076245 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076247 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076250 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076253 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076256 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:44:51.079106 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076258 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076261 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076263 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076266 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076268 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076271 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076273 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076276 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076279 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076281 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076283 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076286 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076288 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076291 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076293 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076296 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076299 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076301 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076304 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076306 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:44:51.079652 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076309 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:44:51.080192 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076312 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:44:51.080192 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076314 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:44:51.080192 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076317 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:44:51.080192 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076321 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:44:51.080192 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076324 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:44:51.080192 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076327 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:44:51.080192 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076330 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:44:51.080192 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076333 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:44:51.080192 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076335 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:44:51.080192 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076338 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:44:51.080192 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076340 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:44:51.080192 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076343 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:44:51.080192 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076345 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:44:51.080192 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076348 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:44:51.080534 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.076352 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 12:44:51.080534 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076471 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:44:51.080534 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076477 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:44:51.080534 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076481 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:44:51.080534 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076484 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:44:51.080534 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076487 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:44:51.080534 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076490 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:44:51.080534 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076492 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:44:51.080534 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076495 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:44:51.080534 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076498 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:44:51.080534 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076500 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:44:51.080534 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076503 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:44:51.080534 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076506 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:44:51.080534 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076508 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076511 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076514 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076533 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076537 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076540 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076543 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076545 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076547 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076550 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076552 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076555 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076557 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076559 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076562 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076564 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076567 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076569 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076572 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076574 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:44:51.080884 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076577 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076579 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076582 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076584 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076586 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076589 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076591 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076593 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076596 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076598 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076601 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076603 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076605 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076608 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076611 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076613 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076616 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076618 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076621 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076623 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:44:51.081362 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076626 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076628 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076630 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076633 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076635 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076638 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076640 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076643 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076645 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076647 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076650 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076652 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076655 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076657 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076659 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076662 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076664 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076666 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076669 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076671 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:44:51.081879 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076674 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076676 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076679 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076681 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076684 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076687 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076695 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076697 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076700 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076702 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076705 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076707 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076710 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076712 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076714 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076717 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076719 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076722 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076725 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:44:51.082340 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076727 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:44:51.082807 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076730 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:44:51.082807 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076732 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:44:51.082807 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076734 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:44:51.082807 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076737 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:44:51.082807 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076739 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:44:51.082807 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076742 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:44:51.082807 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:51.076744 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:44:51.082807 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.076749 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 12:44:51.082807 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.076842 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 12:44:51.082807 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.079759 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 12:44:51.082807 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.081393 2565 server.go:1019] "Starting client certificate rotation" Jul 16 12:44:51.082807 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.081487 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 12:44:51.082807 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.081531 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 12:44:51.084235 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.084222 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 12:44:51.085377 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.085361 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 12:44:51.094912 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.094897 2565 log.go:25] "Validated CRI v1 runtime API" Jul 16 12:44:51.097129 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.097114 2565 log.go:25] "Validated CRI v1 image API" Jul 16 12:44:51.101580 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.101565 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 12:44:51.104879 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.104855 2565 fs.go:135] Filesystem UUIDs: map[6b7baa4c-be83-4bba-9a34-786a9a305976:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e34f8346-72e6-4c31-a2fc-67be2a570c7e:/dev/nvme0n1p4] Jul 16 12:44:51.104933 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.104880 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 12:44:51.111642 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.111541 2565 manager.go:216] Machine: {Timestamp:2026-07-16 12:44:51.109353159 +0000 UTC m=+0.381204073 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3114035 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ef7c71164450ba608631fc8b23191 SystemUUID:ec2ef7c7-1164-450b-a608-631fc8b23191 BootID:479f2c24-6ead-44db-bed7-370dfb975de7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cd:ac:fb:e3:a5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cd:ac:fb:e3:a5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:1f:9c:05:99:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 12:44:51.111642 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.111636 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 12:44:51.111745 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.111705 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 12:44:51.116532 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.116266 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 12:44:51.116670 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.116535 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-198.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 12:44:51.117204 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.117193 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 12:44:51.117240 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.117207 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 12:44:51.117240 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.117215 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 12:44:51.117883 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.117873 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 12:44:51.118960 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.118945 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 12:44:51.119154 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.119144 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 12:44:51.123066 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.123055 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 16 12:44:51.123191 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.123181 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 12:44:51.123220 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.123212 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 16 12:44:51.123258 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.123226 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 12:44:51.123258 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.123248 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 12:44:51.123316 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.123261 2565 kubelet.go:398] "Adding apiserver pod source" Jul 16 12:44:51.123316 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.123272 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 12:44:51.123673 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.123662 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 12:44:51.123715 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.123681 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 12:44:51.125892 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.125878 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 12:44:51.127896 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.127884 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 12:44:51.127937 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.127914 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 12:44:51.128064 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.128048 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 12:44:51.128064 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.128064 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 12:44:51.128173 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.128070 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 12:44:51.128173 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.128075 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 12:44:51.128173 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.128083 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 12:44:51.128173 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.128088 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 12:44:51.128679 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.128668 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 12:44:51.128679 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.128678 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 12:44:51.128783 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.128684 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 12:44:51.128783 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.128689 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 12:44:51.128783 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.128697 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 12:44:51.128783 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.128704 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 12:44:51.130073 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.130063 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 12:44:51.130073 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.130072 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 12:44:51.133016 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.133001 2565 server.go:1268] "Started kubelet" Jul 16 12:44:51.133118 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.133073 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 12:44:51.133174 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.133090 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 12:44:51.133174 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.133156 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 12:44:51.133854 ip-10-0-138-198 systemd[1]: Started Kubernetes Kubelet. Jul 16 12:44:51.135337 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.134357 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-198.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 12:44:51.135460 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.134684 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 12:44:51.135460 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.134794 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 12:44:51.135460 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.135090 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-198.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 12:44:51.135626 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.135615 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 16 12:44:51.137278 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.137251 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 12:44:51.141741 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.140664 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-198.ec2.internal.18c2c5ce0b23eb2c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-198.ec2.internal,UID:ip-10-0-138-198.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-198.ec2.internal,},FirstTimestamp:2026-07-16 12:44:51.132975916 +0000 UTC m=+0.404826814,LastTimestamp:2026-07-16 12:44:51.132975916 +0000 UTC m=+0.404826814,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-198.ec2.internal,}" Jul 16 12:44:51.141886 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.141868 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 12:44:51.142164 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.141991 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 12:44:51.142164 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.142132 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-198.ec2.internal\" not found" Jul 16 12:44:51.143173 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.143155 2565 factory.go:55] Registering systemd factory Jul 16 12:44:51.143173 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.143175 2565 factory.go:223] Registration of the systemd container factory successfully Jul 16 12:44:51.144992 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.143632 2565 factory.go:153] Registering CRI-O factory Jul 16 12:44:51.144992 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.143649 2565 factory.go:223] Registration of the crio container factory successfully Jul 16 12:44:51.144992 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.143701 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 12:44:51.144992 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.143732 2565 factory.go:103] Registering Raw factory Jul 16 12:44:51.144992 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.143745 2565 manager.go:1196] Started watching for new ooms in manager Jul 16 12:44:51.144992 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.143950 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 12:44:51.144992 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.143972 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 12:44:51.144992 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.144053 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 12:44:51.144992 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.144438 2565 manager.go:349] Starting recovery of all containers Jul 16 12:44:51.145542 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.145123 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 16 12:44:51.145542 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.145132 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 16 12:44:51.150244 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.150207 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-198.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 12:44:51.150358 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.150331 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 12:44:51.157132 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.157114 2565 manager.go:354] Recovery completed Jul 16 12:44:51.161220 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.161127 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:44:51.162871 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.162855 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:44:51.162935 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.162883 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:44:51.162935 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.162893 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:44:51.163387 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.163371 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 12:44:51.163387 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.163383 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 12:44:51.163387 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.163386 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:44:51.163579 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.163397 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 16 12:44:51.164001 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.163988 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:44:51.164037 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.164011 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:44:51.164037 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.164020 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:44:51.165057 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.164994 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-198.ec2.internal.18c2c5ce0cec121c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-198.ec2.internal,UID:ip-10-0-138-198.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-198.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-198.ec2.internal,},FirstTimestamp:2026-07-16 12:44:51.1628703 +0000 UTC m=+0.434721198,LastTimestamp:2026-07-16 12:44:51.1628703 +0000 UTC m=+0.434721198,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-198.ec2.internal,}" Jul 16 12:44:51.165508 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.165496 2565 policy_none.go:49] "None policy: Start" Jul 16 12:44:51.165595 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.165513 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 12:44:51.165595 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.165558 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 12:44:51.167477 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.167466 2565 policy_none.go:47] "Start" Jul 16 12:44:51.176392 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.176332 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-198.ec2.internal.18c2c5ce0cec551f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-198.ec2.internal,UID:ip-10-0-138-198.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-198.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-198.ec2.internal,},FirstTimestamp:2026-07-16 12:44:51.162887455 +0000 UTC m=+0.434738353,LastTimestamp:2026-07-16 12:44:51.162887455 +0000 UTC m=+0.434738353,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-198.ec2.internal,}" Jul 16 12:44:51.187640 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.187586 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-198.ec2.internal.18c2c5ce0cec7997 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-198.ec2.internal,UID:ip-10-0-138-198.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-198.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-198.ec2.internal,},FirstTimestamp:2026-07-16 12:44:51.162896791 +0000 UTC m=+0.434747689,LastTimestamp:2026-07-16 12:44:51.162896791 +0000 UTC m=+0.434747689,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-198.ec2.internal,}" Jul 16 12:44:51.188171 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.188153 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bh8nh" Jul 16 12:44:51.199290 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.199211 2565 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-198.ec2.internal.18c2c5ce0cec121c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-198.ec2.internal.18c2c5ce0cec121c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-198.ec2.internal,UID:ip-10-0-138-198.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-198.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-198.ec2.internal,},FirstTimestamp:2026-07-16 12:44:51.1628703 +0000 UTC m=+0.434721198,LastTimestamp:2026-07-16 12:44:51.164000346 +0000 UTC m=+0.435851243,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-198.ec2.internal,}" Jul 16 12:44:51.202044 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.202030 2565 manager.go:337] "Starting Device Plugin manager" Jul 16 12:44:51.202106 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.202060 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 12:44:51.202106 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.202073 2565 server.go:87] "Starting device plugin registration server" Jul 16 12:44:51.202312 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.202297 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 12:44:51.202379 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.202311 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 12:44:51.202431 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.202385 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 12:44:51.202468 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.202462 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 12:44:51.202507 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.202469 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 12:44:51.203238 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.203210 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 12:44:51.203310 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.203298 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-198.ec2.internal\" not found" Jul 16 12:44:51.205132 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.205115 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bh8nh" Jul 16 12:44:51.210441 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.210380 2565 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-198.ec2.internal.18c2c5ce0cec551f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-198.ec2.internal.18c2c5ce0cec551f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-198.ec2.internal,UID:ip-10-0-138-198.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-198.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-198.ec2.internal,},FirstTimestamp:2026-07-16 12:44:51.162887455 +0000 UTC m=+0.434738353,LastTimestamp:2026-07-16 12:44:51.164015526 +0000 UTC m=+0.435866424,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-198.ec2.internal,}" Jul 16 12:44:51.260817 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.260751 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 12:44:51.262133 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.262110 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 12:44:51.262212 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.262157 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 12:44:51.262212 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.262180 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 12:44:51.262312 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.262218 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 12:44:51.265077 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.265062 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:44:51.303198 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.303181 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:44:51.305494 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.305478 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:44:51.305591 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.305507 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:44:51.305591 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.305536 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:44:51.305591 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.305565 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.319507 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.319490 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.319568 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.319510 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-198.ec2.internal\": node \"ip-10-0-138-198.ec2.internal\" not found" Jul 16 12:44:51.363005 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.362982 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-198.ec2.internal"] Jul 16 12:44:51.363076 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.363041 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:44:51.364115 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.364102 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:44:51.364179 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.364128 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:44:51.364179 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.364143 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:44:51.366388 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.366376 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:44:51.366546 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.366513 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.366588 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.366565 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:44:51.367085 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.367062 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:44:51.367167 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.367091 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:44:51.367167 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.367071 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:44:51.367167 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.367101 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:44:51.367167 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.367118 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:44:51.367167 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.367129 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:44:51.369235 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.369218 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.369323 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.369242 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:44:51.370296 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.370269 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:44:51.370390 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.370300 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:44:51.370390 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.370314 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:44:51.392015 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.391996 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-198.ec2.internal\" not found" node="ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.396086 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.396072 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-198.ec2.internal\" not found" node="ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.405489 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.405475 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-198.ec2.internal\" not found" Jul 16 12:44:51.446881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.446860 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d696b861a7ee434644a36496693ecd3e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal\" (UID: \"d696b861a7ee434644a36496693ecd3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.446881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.446882 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6594cb49ca46862b517c534a4f67f101-config\") pod \"kube-apiserver-proxy-ip-10-0-138-198.ec2.internal\" (UID: \"6594cb49ca46862b517c534a4f67f101\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.446992 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.446897 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d696b861a7ee434644a36496693ecd3e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal\" (UID: \"d696b861a7ee434644a36496693ecd3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.506447 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.506409 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-198.ec2.internal\" not found" Jul 16 12:44:51.547987 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.547929 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d696b861a7ee434644a36496693ecd3e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal\" (UID: \"d696b861a7ee434644a36496693ecd3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.547987 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.547902 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d696b861a7ee434644a36496693ecd3e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal\" (UID: \"d696b861a7ee434644a36496693ecd3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.547987 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.547984 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6594cb49ca46862b517c534a4f67f101-config\") pod \"kube-apiserver-proxy-ip-10-0-138-198.ec2.internal\" (UID: \"6594cb49ca46862b517c534a4f67f101\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.548132 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.548014 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d696b861a7ee434644a36496693ecd3e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal\" (UID: \"d696b861a7ee434644a36496693ecd3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.548132 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.548032 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d696b861a7ee434644a36496693ecd3e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal\" (UID: \"d696b861a7ee434644a36496693ecd3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.548132 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.548037 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6594cb49ca46862b517c534a4f67f101-config\") pod \"kube-apiserver-proxy-ip-10-0-138-198.ec2.internal\" (UID: \"6594cb49ca46862b517c534a4f67f101\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.606785 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.606764 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-198.ec2.internal\" not found" Jul 16 12:44:51.693531 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.693486 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.699079 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:51.699064 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-198.ec2.internal" Jul 16 12:44:51.707849 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.707828 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-198.ec2.internal\" not found" Jul 16 12:44:51.808428 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.808366 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-198.ec2.internal\" not found" Jul 16 12:44:51.908996 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:51.908971 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-198.ec2.internal\" not found" Jul 16 12:44:52.009542 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.009509 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-198.ec2.internal\" not found" Jul 16 12:44:52.011489 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.011473 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:44:52.042936 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.042915 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal" Jul 16 12:44:52.063255 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.063207 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 12:44:52.064875 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.064861 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-198.ec2.internal" Jul 16 12:44:52.079387 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.079372 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 12:44:52.080395 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.080382 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 12:44:52.080560 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.080543 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 12:44:52.080617 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.080546 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 12:44:52.123679 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.123662 2565 apiserver.go:52] "Watching apiserver" Jul 16 12:44:52.138971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.138943 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 12:44:52.139279 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.139257 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-zdrc5","kube-system/kube-apiserver-proxy-ip-10-0-138-198.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28","openshift-cluster-node-tuning-operator/tuned-wmjgp","openshift-image-registry/node-ca-g64cf","openshift-multus/multus-7vsds","openshift-multus/network-metrics-daemon-n2c94","openshift-network-operator/iptables-alerter-6dvg9","kube-system/konnectivity-agent-v2l9k","openshift-dns/node-resolver-smgrz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal","openshift-multus/multus-additional-cni-plugins-nv5pc","openshift-network-diagnostics/network-check-target-9w6xv","openshift-ovn-kubernetes/ovnkube-node-x4bdr"] Jul 16 12:44:52.141371 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.141354 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:44:52.141432 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.141414 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:44:52.142152 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.142136 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 12:44:52.143488 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.143464 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-smgrz" Jul 16 12:44:52.145964 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.145826 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r4h5t\"" Jul 16 12:44:52.145964 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.145875 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 12:44:52.146123 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.146072 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 12:44:52.146181 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.146141 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.148408 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.148390 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.148787 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.148763 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 12:44:52.148884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.148799 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 12:44:52.148884 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.148808 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 12:44:52.149001 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.148951 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 12:44:52.149001 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.148969 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 12:44:52.149001 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.148978 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2pc9z\"" Jul 16 12:44:52.149148 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.149113 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 12:44:52.151006 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.150986 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nq4gr\"" Jul 16 12:44:52.151118 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.150992 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 12:44:52.151118 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.150995 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:44:52.151118 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151094 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6dvg9" Jul 16 12:44:52.151382 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151363 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-log-socket\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.151415 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151399 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-cni-netd\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.151447 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151420 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3b63d0-7a24-407c-891b-ca52cb88a27c-tmp-dir\") pod \"node-resolver-smgrz\" (UID: \"eb3b63d0-7a24-407c-891b-ca52cb88a27c\") " pod="openshift-dns/node-resolver-smgrz" Jul 16 12:44:52.151447 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151437 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.151554 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151453 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-cni-bin\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.151554 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151474 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.151554 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151537 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/56aed7de-c338-4c39-98d3-a6e43b58d9b7-ovnkube-script-lib\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.151705 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151584 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb3b63d0-7a24-407c-891b-ca52cb88a27c-hosts-file\") pod \"node-resolver-smgrz\" (UID: \"eb3b63d0-7a24-407c-891b-ca52cb88a27c\") " pod="openshift-dns/node-resolver-smgrz" Jul 16 12:44:52.151705 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151614 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6q9n\" (UniqueName: \"kubernetes.io/projected/eb3b63d0-7a24-407c-891b-ca52cb88a27c-kube-api-access-f6q9n\") pod \"node-resolver-smgrz\" (UID: \"eb3b63d0-7a24-407c-891b-ca52cb88a27c\") " pod="openshift-dns/node-resolver-smgrz" Jul 16 12:44:52.151705 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151642 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-var-lib-openvswitch\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.151705 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/56aed7de-c338-4c39-98d3-a6e43b58d9b7-ovnkube-config\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.151705 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151691 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/56aed7de-c338-4c39-98d3-a6e43b58d9b7-env-overrides\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.151933 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151715 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/56aed7de-c338-4c39-98d3-a6e43b58d9b7-ovn-node-metrics-cert\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.151933 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151739 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpx5t\" (UniqueName: \"kubernetes.io/projected/56aed7de-c338-4c39-98d3-a6e43b58d9b7-kube-api-access-cpx5t\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.151933 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151766 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-kubelet\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.151933 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151789 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-systemd-units\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.151933 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151812 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-slash\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.151933 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151834 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-run-netns\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.151933 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151860 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-node-log\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.151933 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151883 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs\") pod \"network-metrics-daemon-n2c94\" (UID: \"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe\") " pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:44:52.151933 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151906 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqbbr\" (UniqueName: \"kubernetes.io/projected/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-kube-api-access-pqbbr\") pod \"network-metrics-daemon-n2c94\" (UID: \"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe\") " pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:44:52.151933 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151932 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-run-systemd\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.152249 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151955 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-etc-openvswitch\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.152249 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.151979 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-run-openvswitch\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.152249 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.152004 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-run-ovn\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.153136 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.153120 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:44:52.153353 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.153339 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 12:44:52.153482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.153467 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rgqqm\"" Jul 16 12:44:52.153601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.153584 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 12:44:52.153667 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.153600 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.155602 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.155585 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 12:44:52.155666 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.155658 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 12:44:52.155920 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.155907 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-k2vpk\"" Jul 16 12:44:52.156034 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.156021 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 12:44:52.156149 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.156136 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 12:44:52.156239 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.156226 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v2l9k" Jul 16 12:44:52.158280 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.158267 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 12:44:52.158369 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.158356 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w4fcj\"" Jul 16 12:44:52.158480 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.158467 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 12:44:52.158792 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.158779 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:44:52.158842 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.158825 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:44:52.160986 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.160959 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.163295 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.163271 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g64cf" Jul 16 12:44:52.163730 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.163712 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 12:44:52.163827 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.163719 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mkljl\"" Jul 16 12:44:52.163827 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.163720 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 12:44:52.165608 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.165326 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tztl2\"" Jul 16 12:44:52.165608 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.165326 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 12:44:52.165608 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.165583 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 12:44:52.165847 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.165833 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 12:44:52.166961 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.166948 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:44:52.167045 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.166997 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:44:52.169677 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.169648 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.171136 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.171118 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 12:44:52.172065 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.172051 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 12:44:52.172187 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.172172 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 12:44:52.172446 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.172432 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 12:44:52.172665 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.172647 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t65lt\"" Jul 16 12:44:52.197631 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.197609 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mlb4w" Jul 16 12:44:52.205451 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.205430 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mlb4w" Jul 16 12:44:52.211016 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.210968 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 12:39:52 +0000 UTC" deadline="2028-04-24 03:22:45.123725158 +0000 UTC" Jul 16 12:44:52.211016 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.211016 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15542h37m52.912712415s" Jul 16 12:44:52.244895 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.244881 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 12:44:52.252998 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.252975 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-multus-cni-dir\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.253070 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253014 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c403d1e-9431-418e-a96a-270a9608f7ec-multus-daemon-config\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.253070 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253037 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-cnibin\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.253180 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.253180 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253142 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-etc-openvswitch\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.253180 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253166 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-sysctl-conf\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.253342 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253189 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-host\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.253342 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253201 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-etc-openvswitch\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.253342 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253214 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-run-netns\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.253342 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253240 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.253342 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253309 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-kubelet-config\") pod \"global-pull-secret-syncer-zdrc5\" (UID: \"5de8db0d-cb13-482f-b0f6-8259d3cb37f9\") " pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:44:52.253590 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253357 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret\") pod \"global-pull-secret-syncer-zdrc5\" (UID: \"5de8db0d-cb13-482f-b0f6-8259d3cb37f9\") " pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:44:52.253590 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253393 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-socket-dir\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.253590 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3b63d0-7a24-407c-891b-ca52cb88a27c-tmp-dir\") pod \"node-resolver-smgrz\" (UID: \"eb3b63d0-7a24-407c-891b-ca52cb88a27c\") " pod="openshift-dns/node-resolver-smgrz" Jul 16 12:44:52.253590 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253453 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-run-k8s-cni-cncf-io\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.253590 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253479 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-var-lib-cni-bin\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.253590 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253500 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-dbus\") pod \"global-pull-secret-syncer-zdrc5\" (UID: \"5de8db0d-cb13-482f-b0f6-8259d3cb37f9\") " pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:44:52.253590 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/56aed7de-c338-4c39-98d3-a6e43b58d9b7-ovn-node-metrics-cert\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.253857 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253656 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.253857 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253684 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-sys-fs\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.253857 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253706 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc54t\" (UniqueName: \"kubernetes.io/projected/fbe4dca5-fc0f-4607-b057-73a78b901b15-kube-api-access-xc54t\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.253857 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3b63d0-7a24-407c-891b-ca52cb88a27c-tmp-dir\") pod \"node-resolver-smgrz\" (UID: \"eb3b63d0-7a24-407c-891b-ca52cb88a27c\") " pod="openshift-dns/node-resolver-smgrz" Jul 16 12:44:52.253857 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253724 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-run-ovn\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.253857 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253747 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-cni-netd\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.253857 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253772 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-hostroot\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.253857 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-registration-dir\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.253857 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253801 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-run-ovn\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.253857 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253814 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-cni-bin\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.253857 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253835 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-run\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.253857 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253852 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-run-multus-certs\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.253857 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253849 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253861 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-cni-bin\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253866 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1549474-0299-489e-af96-767bdf28937a-host\") pod \"node-ca-g64cf\" (UID: \"e1549474-0299-489e-af96-767bdf28937a\") " pod="openshift-image-registry/node-ca-g64cf" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253851 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-cni-netd\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253905 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6q9n\" (UniqueName: \"kubernetes.io/projected/eb3b63d0-7a24-407c-891b-ca52cb88a27c-kube-api-access-f6q9n\") pod \"node-resolver-smgrz\" (UID: \"eb3b63d0-7a24-407c-891b-ca52cb88a27c\") " pod="openshift-dns/node-resolver-smgrz" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.253935 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-var-lib-kubelet\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254065 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34601419-53fc-4175-b4b9-44de20f22277-konnectivity-ca\") pod \"konnectivity-agent-v2l9k\" (UID: \"34601419-53fc-4175-b4b9-44de20f22277\") " pod="kube-system/konnectivity-agent-v2l9k" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254146 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/56aed7de-c338-4c39-98d3-a6e43b58d9b7-ovnkube-config\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254195 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/56aed7de-c338-4c39-98d3-a6e43b58d9b7-env-overrides\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-kubelet\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254238 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-systemd-units\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254283 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-kubelet\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254296 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-systemd-units\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254315 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-slash\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254338 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-sysconfig\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254359 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs\") pod \"network-metrics-daemon-n2c94\" (UID: \"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe\") " pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254380 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-run-systemd\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.254463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254381 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-slash\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.255125 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254402 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-log-socket\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.255125 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254424 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10fa3856-33f2-4a83-9bcd-4715bdfe96b3-host-slash\") pod \"iptables-alerter-6dvg9\" (UID: \"10fa3856-33f2-4a83-9bcd-4715bdfe96b3\") " pod="openshift-network-operator/iptables-alerter-6dvg9" Jul 16 12:44:52.255125 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254431 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-run-systemd\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.255125 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254445 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmk6j\" (UniqueName: \"kubernetes.io/projected/10fa3856-33f2-4a83-9bcd-4715bdfe96b3-kube-api-access-hmk6j\") pod \"iptables-alerter-6dvg9\" (UID: \"10fa3856-33f2-4a83-9bcd-4715bdfe96b3\") " pod="openshift-network-operator/iptables-alerter-6dvg9" Jul 16 12:44:52.255125 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254466 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-multus-socket-dir-parent\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.255125 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.254473 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:44:52.255125 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254475 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-log-socket\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.255125 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254486 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-system-cni-dir\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.255125 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254508 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fbw8\" (UniqueName: \"kubernetes.io/projected/e1549474-0299-489e-af96-767bdf28937a-kube-api-access-2fbw8\") pod \"node-ca-g64cf\" (UID: \"e1549474-0299-489e-af96-767bdf28937a\") " pod="openshift-image-registry/node-ca-g64cf" Jul 16 12:44:52.255125 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.254567 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs podName:dd3f2e5f-9499-4947-8d80-160b3ffd0bbe nodeName:}" failed. No retries permitted until 2026-07-16 12:44:52.754513691 +0000 UTC m=+2.026364605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs") pod "network-metrics-daemon-n2c94" (UID: "dd3f2e5f-9499-4947-8d80-160b3ffd0bbe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:44:52.255125 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254594 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-var-lib-cni-multus\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.255125 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254612 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/56aed7de-c338-4c39-98d3-a6e43b58d9b7-env-overrides\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.255125 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254614 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jk4z\" (UniqueName: \"kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z\") pod \"network-check-target-9w6xv\" (UID: \"c4ac2604-e084-4ca8-b7de-bc216ac9d843\") " pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:44:52.255125 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254651 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34601419-53fc-4175-b4b9-44de20f22277-agent-certs\") pod \"konnectivity-agent-v2l9k\" (UID: \"34601419-53fc-4175-b4b9-44de20f22277\") " pod="kube-system/konnectivity-agent-v2l9k" Jul 16 12:44:52.255125 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254675 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1549474-0299-489e-af96-767bdf28937a-serviceca\") pod \"node-ca-g64cf\" (UID: \"e1549474-0299-489e-af96-767bdf28937a\") " pod="openshift-image-registry/node-ca-g64cf" Jul 16 12:44:52.255125 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254699 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254724 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-device-dir\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-var-lib-openvswitch\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254752 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/56aed7de-c338-4c39-98d3-a6e43b58d9b7-ovnkube-config\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-kubernetes\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254797 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-os-release\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254809 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-var-lib-openvswitch\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254821 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-var-lib-kubelet\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-run-netns\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254888 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-node-log\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254892 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-run-netns\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254918 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-cnibin\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254935 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54f5221d-4232-4b00-9970-32ae1b48e13e-tmp\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254944 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-node-log\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254951 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45xtb\" (UniqueName: \"kubernetes.io/projected/54f5221d-4232-4b00-9970-32ae1b48e13e-kube-api-access-45xtb\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.254989 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10fa3856-33f2-4a83-9bcd-4715bdfe96b3-iptables-alerter-script\") pod \"iptables-alerter-6dvg9\" (UID: \"10fa3856-33f2-4a83-9bcd-4715bdfe96b3\") " pod="openshift-network-operator/iptables-alerter-6dvg9" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255005 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-system-cni-dir\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.255810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255020 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c403d1e-9431-418e-a96a-270a9608f7ec-cni-binary-copy\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255036 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255052 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/56aed7de-c338-4c39-98d3-a6e43b58d9b7-ovnkube-script-lib\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255119 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255153 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqbbr\" (UniqueName: \"kubernetes.io/projected/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-kube-api-access-pqbbr\") pod \"network-metrics-daemon-n2c94\" (UID: \"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe\") " pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255204 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255231 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-modprobe-d\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255256 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-tuned\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255264 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-run-openvswitch\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255303 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-systemd\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255340 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56aed7de-c338-4c39-98d3-a6e43b58d9b7-run-openvswitch\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255370 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-lib-modules\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255394 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-multus-conf-dir\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255418 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9f65\" (UniqueName: \"kubernetes.io/projected/3c403d1e-9431-418e-a96a-270a9608f7ec-kube-api-access-x9f65\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255445 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb3b63d0-7a24-407c-891b-ca52cb88a27c-hosts-file\") pod \"node-resolver-smgrz\" (UID: \"eb3b63d0-7a24-407c-891b-ca52cb88a27c\") " pod="openshift-dns/node-resolver-smgrz" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255474 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-sysctl-d\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.256339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255487 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb3b63d0-7a24-407c-891b-ca52cb88a27c-hosts-file\") pod \"node-resolver-smgrz\" (UID: \"eb3b63d0-7a24-407c-891b-ca52cb88a27c\") " pod="openshift-dns/node-resolver-smgrz" Jul 16 12:44:52.257005 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255490 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-sys\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.257005 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255541 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-etc-kubernetes\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.257005 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255568 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-os-release\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.257005 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255594 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-cni-binary-copy\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.257005 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255613 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/56aed7de-c338-4c39-98d3-a6e43b58d9b7-ovnkube-script-lib\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.257005 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255617 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.257005 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255670 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq4dn\" (UniqueName: \"kubernetes.io/projected/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-kube-api-access-rq4dn\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.257005 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.255698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpx5t\" (UniqueName: \"kubernetes.io/projected/56aed7de-c338-4c39-98d3-a6e43b58d9b7-kube-api-access-cpx5t\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.257577 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.257558 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/56aed7de-c338-4c39-98d3-a6e43b58d9b7-ovn-node-metrics-cert\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.284406 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.284384 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpx5t\" (UniqueName: \"kubernetes.io/projected/56aed7de-c338-4c39-98d3-a6e43b58d9b7-kube-api-access-cpx5t\") pod \"ovnkube-node-x4bdr\" (UID: \"56aed7de-c338-4c39-98d3-a6e43b58d9b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.284486 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.284459 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6q9n\" (UniqueName: \"kubernetes.io/projected/eb3b63d0-7a24-407c-891b-ca52cb88a27c-kube-api-access-f6q9n\") pod \"node-resolver-smgrz\" (UID: \"eb3b63d0-7a24-407c-891b-ca52cb88a27c\") " pod="openshift-dns/node-resolver-smgrz" Jul 16 12:44:52.284552 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.284507 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqbbr\" (UniqueName: \"kubernetes.io/projected/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-kube-api-access-pqbbr\") pod \"network-metrics-daemon-n2c94\" (UID: \"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe\") " pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:44:52.321875 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:52.321809 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6594cb49ca46862b517c534a4f67f101.slice/crio-b2ad222113324886f8feeefcc099bc8a12b0ce6e6718a1e42bf7c4b472e04433 WatchSource:0}: Error finding container b2ad222113324886f8feeefcc099bc8a12b0ce6e6718a1e42bf7c4b472e04433: Status 404 returned error can't find the container with id b2ad222113324886f8feeefcc099bc8a12b0ce6e6718a1e42bf7c4b472e04433 Jul 16 12:44:52.322277 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:52.322259 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd696b861a7ee434644a36496693ecd3e.slice/crio-82f8de886ea6b18b5e006b015b684077c47bcce4696edd8f9da6b282c86c9a1c WatchSource:0}: Error finding container 82f8de886ea6b18b5e006b015b684077c47bcce4696edd8f9da6b282c86c9a1c: Status 404 returned error can't find the container with id 82f8de886ea6b18b5e006b015b684077c47bcce4696edd8f9da6b282c86c9a1c Jul 16 12:44:52.328779 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.328764 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:44:52.356342 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356320 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-sysconfig\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.356423 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356353 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10fa3856-33f2-4a83-9bcd-4715bdfe96b3-host-slash\") pod \"iptables-alerter-6dvg9\" (UID: \"10fa3856-33f2-4a83-9bcd-4715bdfe96b3\") " pod="openshift-network-operator/iptables-alerter-6dvg9" Jul 16 12:44:52.356423 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356378 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmk6j\" (UniqueName: \"kubernetes.io/projected/10fa3856-33f2-4a83-9bcd-4715bdfe96b3-kube-api-access-hmk6j\") pod \"iptables-alerter-6dvg9\" (UID: \"10fa3856-33f2-4a83-9bcd-4715bdfe96b3\") " pod="openshift-network-operator/iptables-alerter-6dvg9" Jul 16 12:44:52.356492 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356437 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-sysconfig\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.356492 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356451 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10fa3856-33f2-4a83-9bcd-4715bdfe96b3-host-slash\") pod \"iptables-alerter-6dvg9\" (UID: \"10fa3856-33f2-4a83-9bcd-4715bdfe96b3\") " pod="openshift-network-operator/iptables-alerter-6dvg9" Jul 16 12:44:52.356492 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356481 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-multus-socket-dir-parent\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.356629 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356503 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-system-cni-dir\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.356629 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356535 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fbw8\" (UniqueName: \"kubernetes.io/projected/e1549474-0299-489e-af96-767bdf28937a-kube-api-access-2fbw8\") pod \"node-ca-g64cf\" (UID: \"e1549474-0299-489e-af96-767bdf28937a\") " pod="openshift-image-registry/node-ca-g64cf" Jul 16 12:44:52.356629 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356559 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-var-lib-cni-multus\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.356629 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356589 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-multus-socket-dir-parent\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.356629 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356600 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-system-cni-dir\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.356629 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356620 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-var-lib-cni-multus\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.356867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jk4z\" (UniqueName: \"kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z\") pod \"network-check-target-9w6xv\" (UID: \"c4ac2604-e084-4ca8-b7de-bc216ac9d843\") " pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:44:52.356867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356662 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34601419-53fc-4175-b4b9-44de20f22277-agent-certs\") pod \"konnectivity-agent-v2l9k\" (UID: \"34601419-53fc-4175-b4b9-44de20f22277\") " pod="kube-system/konnectivity-agent-v2l9k" Jul 16 12:44:52.356867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1549474-0299-489e-af96-767bdf28937a-serviceca\") pod \"node-ca-g64cf\" (UID: \"e1549474-0299-489e-af96-767bdf28937a\") " pod="openshift-image-registry/node-ca-g64cf" Jul 16 12:44:52.356867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.356867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356713 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-device-dir\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.356867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356731 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-kubernetes\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.356867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-os-release\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.356867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356784 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.356867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356800 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-os-release\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.356867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356811 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-kubernetes\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.356867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356790 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-device-dir\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.356867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356840 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-var-lib-kubelet\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356903 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-cnibin\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356952 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-var-lib-kubelet\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356963 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-cnibin\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.356988 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54f5221d-4232-4b00-9970-32ae1b48e13e-tmp\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45xtb\" (UniqueName: \"kubernetes.io/projected/54f5221d-4232-4b00-9970-32ae1b48e13e-kube-api-access-45xtb\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357029 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10fa3856-33f2-4a83-9bcd-4715bdfe96b3-iptables-alerter-script\") pod \"iptables-alerter-6dvg9\" (UID: \"10fa3856-33f2-4a83-9bcd-4715bdfe96b3\") " pod="openshift-network-operator/iptables-alerter-6dvg9" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357044 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-system-cni-dir\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357065 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c403d1e-9431-418e-a96a-270a9608f7ec-cni-binary-copy\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357090 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-modprobe-d\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357111 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-tuned\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357135 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-systemd\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-lib-modules\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357163 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-system-cni-dir\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357179 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-multus-conf-dir\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357175 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1549474-0299-489e-af96-767bdf28937a-serviceca\") pod \"node-ca-g64cf\" (UID: \"e1549474-0299-489e-af96-767bdf28937a\") " pod="openshift-image-registry/node-ca-g64cf" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357203 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9f65\" (UniqueName: \"kubernetes.io/projected/3c403d1e-9431-418e-a96a-270a9608f7ec-kube-api-access-x9f65\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-sysctl-d\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.357378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357246 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-sys\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357281 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-systemd\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357539 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-multus-conf-dir\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357687 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-sysctl-d\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357773 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-modprobe-d\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357784 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10fa3856-33f2-4a83-9bcd-4715bdfe96b3-iptables-alerter-script\") pod \"iptables-alerter-6dvg9\" (UID: \"10fa3856-33f2-4a83-9bcd-4715bdfe96b3\") " pod="openshift-network-operator/iptables-alerter-6dvg9" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357799 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c403d1e-9431-418e-a96a-270a9608f7ec-cni-binary-copy\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357848 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-etc-kubernetes\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357876 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-os-release\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357920 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-lib-modules\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357923 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-cni-binary-copy\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357940 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-sys\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357969 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.357996 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4dn\" (UniqueName: \"kubernetes.io/projected/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-kube-api-access-rq4dn\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358026 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-etc-kubernetes\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358053 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-os-release\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358027 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-multus-cni-dir\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.358231 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358101 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c403d1e-9431-418e-a96a-270a9608f7ec-multus-daemon-config\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-cnibin\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358135 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-multus-cni-dir\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358149 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358176 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-sysctl-conf\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358186 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-cnibin\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-host\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358224 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-run-netns\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358251 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358257 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-host\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358270 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358277 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-kubelet-config\") pod \"global-pull-secret-syncer-zdrc5\" (UID: \"5de8db0d-cb13-482f-b0f6-8259d3cb37f9\") " pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358296 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-sysctl-conf\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358301 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret\") pod \"global-pull-secret-syncer-zdrc5\" (UID: \"5de8db0d-cb13-482f-b0f6-8259d3cb37f9\") " pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358314 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-run-netns\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358324 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-socket-dir\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358350 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-run-k8s-cni-cncf-io\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.359025 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358369 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-kubelet-config\") pod \"global-pull-secret-syncer-zdrc5\" (UID: \"5de8db0d-cb13-482f-b0f6-8259d3cb37f9\") " pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358379 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-var-lib-cni-bin\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358405 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-dbus\") pod \"global-pull-secret-syncer-zdrc5\" (UID: \"5de8db0d-cb13-482f-b0f6-8259d3cb37f9\") " pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358431 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358452 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-sys-fs\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358477 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc54t\" (UniqueName: \"kubernetes.io/projected/fbe4dca5-fc0f-4607-b057-73a78b901b15-kube-api-access-xc54t\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358466 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-socket-dir\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358492 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-cni-binary-copy\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358536 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-hostroot\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358563 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-registration-dir\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358569 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c403d1e-9431-418e-a96a-270a9608f7ec-multus-daemon-config\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-run\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358613 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-run-multus-certs\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358621 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-hostroot\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1549474-0299-489e-af96-767bdf28937a-host\") pod \"node-ca-g64cf\" (UID: \"e1549474-0299-489e-af96-767bdf28937a\") " pod="openshift-image-registry/node-ca-g64cf" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358632 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-dbus\") pod \"global-pull-secret-syncer-zdrc5\" (UID: \"5de8db0d-cb13-482f-b0f6-8259d3cb37f9\") " pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358686 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-run-multus-certs\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358692 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-run-k8s-cni-cncf-io\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.359971 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.358694 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:44:52.360711 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358738 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-registration-dir\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.360711 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358755 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbe4dca5-fc0f-4607-b057-73a78b901b15-sys-fs\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.360711 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.358744 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret podName:5de8db0d-cb13-482f-b0f6-8259d3cb37f9 nodeName:}" failed. No retries permitted until 2026-07-16 12:44:52.858727791 +0000 UTC m=+2.130578701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret") pod "global-pull-secret-syncer-zdrc5" (UID: "5de8db0d-cb13-482f-b0f6-8259d3cb37f9") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:44:52.360711 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358769 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.360711 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358787 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1549474-0299-489e-af96-767bdf28937a-host\") pod \"node-ca-g64cf\" (UID: \"e1549474-0299-489e-af96-767bdf28937a\") " pod="openshift-image-registry/node-ca-g64cf" Jul 16 12:44:52.360711 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358789 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-var-lib-kubelet\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.360711 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34601419-53fc-4175-b4b9-44de20f22277-konnectivity-ca\") pod \"konnectivity-agent-v2l9k\" (UID: \"34601419-53fc-4175-b4b9-44de20f22277\") " pod="kube-system/konnectivity-agent-v2l9k" Jul 16 12:44:52.360711 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358823 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c403d1e-9431-418e-a96a-270a9608f7ec-host-var-lib-cni-bin\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.360711 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358865 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-var-lib-kubelet\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.360711 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.358921 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/54f5221d-4232-4b00-9970-32ae1b48e13e-run\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.360711 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.359018 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.360711 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.359163 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.360711 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.359557 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34601419-53fc-4175-b4b9-44de20f22277-agent-certs\") pod \"konnectivity-agent-v2l9k\" (UID: \"34601419-53fc-4175-b4b9-44de20f22277\") " pod="kube-system/konnectivity-agent-v2l9k" Jul 16 12:44:52.360711 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.359876 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34601419-53fc-4175-b4b9-44de20f22277-konnectivity-ca\") pod \"konnectivity-agent-v2l9k\" (UID: \"34601419-53fc-4175-b4b9-44de20f22277\") " pod="kube-system/konnectivity-agent-v2l9k" Jul 16 12:44:52.360711 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.360463 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/54f5221d-4232-4b00-9970-32ae1b48e13e-etc-tuned\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.360711 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.360646 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54f5221d-4232-4b00-9970-32ae1b48e13e-tmp\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.372347 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.372327 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:44:52.372347 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.372350 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:44:52.372555 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.372364 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8jk4z for pod openshift-network-diagnostics/network-check-target-9w6xv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:44:52.372555 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.372464 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z podName:c4ac2604-e084-4ca8-b7de-bc216ac9d843 nodeName:}" failed. No retries permitted until 2026-07-16 12:44:52.872437931 +0000 UTC m=+2.144288838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8jk4z" (UniqueName: "kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z") pod "network-check-target-9w6xv" (UID: "c4ac2604-e084-4ca8-b7de-bc216ac9d843") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:44:52.374642 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.374180 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmk6j\" (UniqueName: \"kubernetes.io/projected/10fa3856-33f2-4a83-9bcd-4715bdfe96b3-kube-api-access-hmk6j\") pod \"iptables-alerter-6dvg9\" (UID: \"10fa3856-33f2-4a83-9bcd-4715bdfe96b3\") " pod="openshift-network-operator/iptables-alerter-6dvg9" Jul 16 12:44:52.374642 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.374356 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45xtb\" (UniqueName: \"kubernetes.io/projected/54f5221d-4232-4b00-9970-32ae1b48e13e-kube-api-access-45xtb\") pod \"tuned-wmjgp\" (UID: \"54f5221d-4232-4b00-9970-32ae1b48e13e\") " pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.375178 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.375149 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc54t\" (UniqueName: \"kubernetes.io/projected/fbe4dca5-fc0f-4607-b057-73a78b901b15-kube-api-access-xc54t\") pod \"aws-ebs-csi-driver-node-mqr28\" (UID: \"fbe4dca5-fc0f-4607-b057-73a78b901b15\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.375660 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.375641 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq4dn\" (UniqueName: \"kubernetes.io/projected/d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac-kube-api-access-rq4dn\") pod \"multus-additional-cni-plugins-nv5pc\" (UID: \"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac\") " pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.377536 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.377501 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9f65\" (UniqueName: \"kubernetes.io/projected/3c403d1e-9431-418e-a96a-270a9608f7ec-kube-api-access-x9f65\") pod \"multus-7vsds\" (UID: \"3c403d1e-9431-418e-a96a-270a9608f7ec\") " pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.377627 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.377589 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fbw8\" (UniqueName: \"kubernetes.io/projected/e1549474-0299-489e-af96-767bdf28937a-kube-api-access-2fbw8\") pod \"node-ca-g64cf\" (UID: \"e1549474-0299-489e-af96-767bdf28937a\") " pod="openshift-image-registry/node-ca-g64cf" Jul 16 12:44:52.468809 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.468787 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-smgrz" Jul 16 12:44:52.478221 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:52.478191 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3b63d0_7a24_407c_891b_ca52cb88a27c.slice/crio-2dff2f3556ad89fb322635267c4d8803f7e2f5135e646d819a1ff2e4c26c9b24 WatchSource:0}: Error finding container 2dff2f3556ad89fb322635267c4d8803f7e2f5135e646d819a1ff2e4c26c9b24: Status 404 returned error can't find the container with id 2dff2f3556ad89fb322635267c4d8803f7e2f5135e646d819a1ff2e4c26c9b24 Jul 16 12:44:52.485211 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.485191 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:44:52.493129 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:52.493106 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56aed7de_c338_4c39_98d3_a6e43b58d9b7.slice/crio-6847dfe156952822ad0df1173e9cc48a928b85eade0c55d0419a1629c1acf297 WatchSource:0}: Error finding container 6847dfe156952822ad0df1173e9cc48a928b85eade0c55d0419a1629c1acf297: Status 404 returned error can't find the container with id 6847dfe156952822ad0df1173e9cc48a928b85eade0c55d0419a1629c1acf297 Jul 16 12:44:52.507487 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.507469 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" Jul 16 12:44:52.512830 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.512815 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:44:52.513596 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.513570 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6dvg9" Jul 16 12:44:52.515417 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:52.515397 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f5221d_4232_4b00_9970_32ae1b48e13e.slice/crio-9016b70b44c0c27aa6ccab3dea4accb32c8a76056cc18a3d029be97f3689ea99 WatchSource:0}: Error finding container 9016b70b44c0c27aa6ccab3dea4accb32c8a76056cc18a3d029be97f3689ea99: Status 404 returned error can't find the container with id 9016b70b44c0c27aa6ccab3dea4accb32c8a76056cc18a3d029be97f3689ea99 Jul 16 12:44:52.536616 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.536598 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7vsds" Jul 16 12:44:52.545276 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:52.545257 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c403d1e_9431_418e_a96a_270a9608f7ec.slice/crio-7b272b08f9e4b18e8180d249f8990bf6c75efc0dfa8e6f14b9acdcb7954bda95 WatchSource:0}: Error finding container 7b272b08f9e4b18e8180d249f8990bf6c75efc0dfa8e6f14b9acdcb7954bda95: Status 404 returned error can't find the container with id 7b272b08f9e4b18e8180d249f8990bf6c75efc0dfa8e6f14b9acdcb7954bda95 Jul 16 12:44:52.553599 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.553584 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v2l9k" Jul 16 12:44:52.560803 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:52.560785 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34601419_53fc_4175_b4b9_44de20f22277.slice/crio-44c488e2d7a93a8ed5e168ef7eea6041749a3d6df3c611cf1effa5b906a49a19 WatchSource:0}: Error finding container 44c488e2d7a93a8ed5e168ef7eea6041749a3d6df3c611cf1effa5b906a49a19: Status 404 returned error can't find the container with id 44c488e2d7a93a8ed5e168ef7eea6041749a3d6df3c611cf1effa5b906a49a19 Jul 16 12:44:52.569032 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.569013 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nv5pc" Jul 16 12:44:52.577433 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:52.577413 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08b5ee1_e4dc_4f6c_b2a9_d8b6d560eaac.slice/crio-8b86b256b81971db3ad8c3d604a35c20bc68a17e1cdb557b9370c65da0dd7e5c WatchSource:0}: Error finding container 8b86b256b81971db3ad8c3d604a35c20bc68a17e1cdb557b9370c65da0dd7e5c: Status 404 returned error can't find the container with id 8b86b256b81971db3ad8c3d604a35c20bc68a17e1cdb557b9370c65da0dd7e5c Jul 16 12:44:52.584289 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.584273 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g64cf" Jul 16 12:44:52.592023 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.592006 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" Jul 16 12:44:52.593207 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:52.593188 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1549474_0299_489e_af96_767bdf28937a.slice/crio-7870430a1141c8ac8bb41e3b01312bbc6eb0f6c045dacbf95c7b22c05b6a01b4 WatchSource:0}: Error finding container 7870430a1141c8ac8bb41e3b01312bbc6eb0f6c045dacbf95c7b22c05b6a01b4: Status 404 returned error can't find the container with id 7870430a1141c8ac8bb41e3b01312bbc6eb0f6c045dacbf95c7b22c05b6a01b4 Jul 16 12:44:52.601373 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:44:52.601352 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe4dca5_fc0f_4607_b057_73a78b901b15.slice/crio-81e4834e6ead89a172f49db998b05a7b11c65e1a87ce6acffef73e0b92710dc9 WatchSource:0}: Error finding container 81e4834e6ead89a172f49db998b05a7b11c65e1a87ce6acffef73e0b92710dc9: Status 404 returned error can't find the container with id 81e4834e6ead89a172f49db998b05a7b11c65e1a87ce6acffef73e0b92710dc9 Jul 16 12:44:52.610111 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.610091 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:44:52.761624 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.761584 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs\") pod \"network-metrics-daemon-n2c94\" (UID: \"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe\") " pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:44:52.761804 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.761781 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:44:52.761869 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.761857 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs podName:dd3f2e5f-9499-4947-8d80-160b3ffd0bbe nodeName:}" failed. No retries permitted until 2026-07-16 12:44:53.761839384 +0000 UTC m=+3.033690269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs") pod "network-metrics-daemon-n2c94" (UID: "dd3f2e5f-9499-4947-8d80-160b3ffd0bbe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:44:52.862350 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.862279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret\") pod \"global-pull-secret-syncer-zdrc5\" (UID: \"5de8db0d-cb13-482f-b0f6-8259d3cb37f9\") " pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:44:52.862483 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.862417 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:44:52.862547 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.862489 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret podName:5de8db0d-cb13-482f-b0f6-8259d3cb37f9 nodeName:}" failed. No retries permitted until 2026-07-16 12:44:53.862470032 +0000 UTC m=+3.134320936 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret") pod "global-pull-secret-syncer-zdrc5" (UID: "5de8db0d-cb13-482f-b0f6-8259d3cb37f9") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:44:52.963069 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:52.963029 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jk4z\" (UniqueName: \"kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z\") pod \"network-check-target-9w6xv\" (UID: \"c4ac2604-e084-4ca8-b7de-bc216ac9d843\") " pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:44:52.963241 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.963222 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:44:52.963241 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.963241 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:44:52.963348 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.963254 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8jk4z for pod openshift-network-diagnostics/network-check-target-9w6xv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:44:52.963348 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:52.963308 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z podName:c4ac2604-e084-4ca8-b7de-bc216ac9d843 nodeName:}" failed. No retries permitted until 2026-07-16 12:44:53.963291051 +0000 UTC m=+3.235141953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jk4z" (UniqueName: "kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z") pod "network-check-target-9w6xv" (UID: "c4ac2604-e084-4ca8-b7de-bc216ac9d843") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:44:54.485927 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:54.485837 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 12:39:53 +0000 UTC" deadline="2027-12-24 16:48:13.281350431 +0000 UTC" Jul 16 12:44:54.485927 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:54.485871 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12628h3m18.795484067s" Jul 16 12:44:54.559057 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:54.558913 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6dvg9" event={"ID":"10fa3856-33f2-4a83-9bcd-4715bdfe96b3","Type":"ContainerStarted","Data":"da578a9e8d17832858b5846c4be97ed3cd89c036175abd888840c8bd31dde8ee"} Jul 16 12:44:54.576676 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:54.576637 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-smgrz" event={"ID":"eb3b63d0-7a24-407c-891b-ca52cb88a27c","Type":"ContainerStarted","Data":"2dff2f3556ad89fb322635267c4d8803f7e2f5135e646d819a1ff2e4c26c9b24"} Jul 16 12:44:54.580492 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:54.580464 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-198.ec2.internal" event={"ID":"6594cb49ca46862b517c534a4f67f101","Type":"ContainerStarted","Data":"b2ad222113324886f8feeefcc099bc8a12b0ce6e6718a1e42bf7c4b472e04433"} Jul 16 12:44:54.588287 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:54.587393 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" event={"ID":"fbe4dca5-fc0f-4607-b057-73a78b901b15","Type":"ContainerStarted","Data":"81e4834e6ead89a172f49db998b05a7b11c65e1a87ce6acffef73e0b92710dc9"} Jul 16 12:44:54.598551 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:54.596816 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" event={"ID":"54f5221d-4232-4b00-9970-32ae1b48e13e","Type":"ContainerStarted","Data":"9016b70b44c0c27aa6ccab3dea4accb32c8a76056cc18a3d029be97f3689ea99"} Jul 16 12:44:54.599049 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:54.598969 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" event={"ID":"56aed7de-c338-4c39-98d3-a6e43b58d9b7","Type":"ContainerStarted","Data":"6847dfe156952822ad0df1173e9cc48a928b85eade0c55d0419a1629c1acf297"} Jul 16 12:44:54.604111 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:54.604071 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal" event={"ID":"d696b861a7ee434644a36496693ecd3e","Type":"ContainerStarted","Data":"82f8de886ea6b18b5e006b015b684077c47bcce4696edd8f9da6b282c86c9a1c"} Jul 16 12:44:54.611934 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:54.611905 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g64cf" event={"ID":"e1549474-0299-489e-af96-767bdf28937a","Type":"ContainerStarted","Data":"7870430a1141c8ac8bb41e3b01312bbc6eb0f6c045dacbf95c7b22c05b6a01b4"} Jul 16 12:44:54.636819 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:54.636618 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nv5pc" event={"ID":"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac","Type":"ContainerStarted","Data":"8b86b256b81971db3ad8c3d604a35c20bc68a17e1cdb557b9370c65da0dd7e5c"} Jul 16 12:44:54.641953 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:54.641846 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v2l9k" event={"ID":"34601419-53fc-4175-b4b9-44de20f22277","Type":"ContainerStarted","Data":"44c488e2d7a93a8ed5e168ef7eea6041749a3d6df3c611cf1effa5b906a49a19"} Jul 16 12:44:54.644734 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:54.644679 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7vsds" event={"ID":"3c403d1e-9431-418e-a96a-270a9608f7ec","Type":"ContainerStarted","Data":"7b272b08f9e4b18e8180d249f8990bf6c75efc0dfa8e6f14b9acdcb7954bda95"} Jul 16 12:44:54.880966 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:54.880887 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:44:55.046677 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:55.046641 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs\") pod \"network-metrics-daemon-n2c94\" (UID: \"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe\") " pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:44:55.046860 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:55.046839 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:44:55.046922 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:55.046905 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs podName:dd3f2e5f-9499-4947-8d80-160b3ffd0bbe nodeName:}" failed. No retries permitted until 2026-07-16 12:44:57.046885408 +0000 UTC m=+5.042026165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs") pod "network-metrics-daemon-n2c94" (UID: "dd3f2e5f-9499-4947-8d80-160b3ffd0bbe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:44:55.147584 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:55.147490 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret\") pod \"global-pull-secret-syncer-zdrc5\" (UID: \"5de8db0d-cb13-482f-b0f6-8259d3cb37f9\") " pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:44:55.147758 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:55.147659 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:44:55.147758 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:55.147719 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret podName:5de8db0d-cb13-482f-b0f6-8259d3cb37f9 nodeName:}" failed. No retries permitted until 2026-07-16 12:44:57.147701994 +0000 UTC m=+5.142842747 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret") pod "global-pull-secret-syncer-zdrc5" (UID: "5de8db0d-cb13-482f-b0f6-8259d3cb37f9") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:44:55.248751 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:55.248709 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jk4z\" (UniqueName: \"kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z\") pod \"network-check-target-9w6xv\" (UID: \"c4ac2604-e084-4ca8-b7de-bc216ac9d843\") " pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:44:55.248989 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:55.248901 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:44:55.248989 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:55.248927 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:44:55.248989 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:55.248940 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8jk4z for pod openshift-network-diagnostics/network-check-target-9w6xv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:44:55.249161 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:55.248997 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z podName:c4ac2604-e084-4ca8-b7de-bc216ac9d843 nodeName:}" failed. No retries permitted until 2026-07-16 12:44:57.248978369 +0000 UTC m=+5.244119126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jk4z" (UniqueName: "kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z") pod "network-check-target-9w6xv" (UID: "c4ac2604-e084-4ca8-b7de-bc216ac9d843") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:44:55.486886 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:55.486726 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 12:39:53 +0000 UTC" deadline="2028-01-29 11:15:04.31276265 +0000 UTC" Jul 16 12:44:55.486886 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:55.486791 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13486h30m8.825976335s" Jul 16 12:44:55.540585 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:55.539733 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:44:55.540585 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:55.539937 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:44:55.540585 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:55.540339 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:44:55.540585 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:55.540423 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:44:55.540585 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:55.540484 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:44:55.540585 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:55.540551 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:44:56.225011 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:56.224978 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:44:57.063378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:57.063292 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs\") pod \"network-metrics-daemon-n2c94\" (UID: \"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe\") " pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:44:57.063823 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:57.063451 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:44:57.063823 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:57.063513 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs podName:dd3f2e5f-9499-4947-8d80-160b3ffd0bbe nodeName:}" failed. No retries permitted until 2026-07-16 12:45:01.063494141 +0000 UTC m=+9.058634899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs") pod "network-metrics-daemon-n2c94" (UID: "dd3f2e5f-9499-4947-8d80-160b3ffd0bbe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:44:57.164728 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:57.164690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret\") pod \"global-pull-secret-syncer-zdrc5\" (UID: \"5de8db0d-cb13-482f-b0f6-8259d3cb37f9\") " pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:44:57.164906 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:57.164844 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:44:57.164961 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:57.164910 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret podName:5de8db0d-cb13-482f-b0f6-8259d3cb37f9 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:01.164891768 +0000 UTC m=+9.160032511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret") pod "global-pull-secret-syncer-zdrc5" (UID: "5de8db0d-cb13-482f-b0f6-8259d3cb37f9") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:44:57.266012 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:57.265425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jk4z\" (UniqueName: \"kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z\") pod \"network-check-target-9w6xv\" (UID: \"c4ac2604-e084-4ca8-b7de-bc216ac9d843\") " pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:44:57.266012 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:57.265595 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:44:57.266012 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:57.265613 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:44:57.266012 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:57.265624 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8jk4z for pod openshift-network-diagnostics/network-check-target-9w6xv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:44:57.266012 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:57.265686 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z podName:c4ac2604-e084-4ca8-b7de-bc216ac9d843 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:01.265669356 +0000 UTC m=+9.260810096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jk4z" (UniqueName: "kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z") pod "network-check-target-9w6xv" (UID: "c4ac2604-e084-4ca8-b7de-bc216ac9d843") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:44:57.540214 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:57.540102 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:44:57.540378 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:57.540218 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:44:57.540378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:57.540297 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:44:57.540378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:57.540297 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:44:57.540546 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:57.540390 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:44:57.540546 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:57.540446 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:44:59.539991 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:59.539955 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:44:59.540454 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:59.540112 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:44:59.540549 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:59.540486 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:44:59.540605 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:59.540577 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:44:59.540656 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:44:59.540625 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:44:59.540724 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:44:59.540702 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:45:01.100215 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:01.100131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs\") pod \"network-metrics-daemon-n2c94\" (UID: \"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe\") " pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:01.100685 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:01.100287 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:01.100685 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:01.100352 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs podName:dd3f2e5f-9499-4947-8d80-160b3ffd0bbe nodeName:}" failed. No retries permitted until 2026-07-16 12:45:09.100333098 +0000 UTC m=+17.095473837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs") pod "network-metrics-daemon-n2c94" (UID: "dd3f2e5f-9499-4947-8d80-160b3ffd0bbe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:01.201452 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:01.201415 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret\") pod \"global-pull-secret-syncer-zdrc5\" (UID: \"5de8db0d-cb13-482f-b0f6-8259d3cb37f9\") " pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:01.201619 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:01.201596 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:01.201708 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:01.201665 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret podName:5de8db0d-cb13-482f-b0f6-8259d3cb37f9 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:09.20164738 +0000 UTC m=+17.196788143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret") pod "global-pull-secret-syncer-zdrc5" (UID: "5de8db0d-cb13-482f-b0f6-8259d3cb37f9") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:01.302851 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:01.302155 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jk4z\" (UniqueName: \"kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z\") pod \"network-check-target-9w6xv\" (UID: \"c4ac2604-e084-4ca8-b7de-bc216ac9d843\") " pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:01.302851 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:01.302395 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:45:01.302851 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:01.302418 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:45:01.302851 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:01.302433 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8jk4z for pod openshift-network-diagnostics/network-check-target-9w6xv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:01.302851 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:01.302519 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z podName:c4ac2604-e084-4ca8-b7de-bc216ac9d843 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:09.302499669 +0000 UTC m=+17.297640443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jk4z" (UniqueName: "kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z") pod "network-check-target-9w6xv" (UID: "c4ac2604-e084-4ca8-b7de-bc216ac9d843") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:01.540224 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:01.539571 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:01.540224 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:01.539681 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:45:01.540224 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:01.539995 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:01.540224 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:01.540061 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:45:01.540224 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:01.540105 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:01.540224 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:01.540150 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:45:03.540225 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:03.539715 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:03.540225 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:03.539754 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:03.540225 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:03.539729 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:03.540225 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:03.539851 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:45:03.540225 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:03.539917 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:45:03.540225 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:03.539977 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:45:05.540068 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:05.539817 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:05.540506 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:05.539857 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:05.540506 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:05.540177 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:45:05.540506 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:05.540299 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:45:05.540506 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:05.539889 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:05.540506 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:05.540394 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:45:07.539258 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:07.539202 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:07.539258 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:07.539218 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:07.539717 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:07.539215 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:07.539717 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:07.539331 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:45:07.539717 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:07.539408 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:45:07.539717 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:07.539469 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:45:09.162917 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:09.162882 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs\") pod \"network-metrics-daemon-n2c94\" (UID: \"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe\") " pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:09.163430 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:09.163045 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:09.163430 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:09.163116 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs podName:dd3f2e5f-9499-4947-8d80-160b3ffd0bbe nodeName:}" failed. No retries permitted until 2026-07-16 12:45:25.163094493 +0000 UTC m=+33.158235234 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs") pod "network-metrics-daemon-n2c94" (UID: "dd3f2e5f-9499-4947-8d80-160b3ffd0bbe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:09.263478 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:09.263450 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret\") pod \"global-pull-secret-syncer-zdrc5\" (UID: \"5de8db0d-cb13-482f-b0f6-8259d3cb37f9\") " pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:09.263624 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:09.263553 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:09.263624 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:09.263607 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret podName:5de8db0d-cb13-482f-b0f6-8259d3cb37f9 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:25.263587196 +0000 UTC m=+33.258727955 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret") pod "global-pull-secret-syncer-zdrc5" (UID: "5de8db0d-cb13-482f-b0f6-8259d3cb37f9") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:09.364661 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:09.364622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jk4z\" (UniqueName: \"kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z\") pod \"network-check-target-9w6xv\" (UID: \"c4ac2604-e084-4ca8-b7de-bc216ac9d843\") " pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:09.364825 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:09.364786 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:45:09.364825 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:09.364807 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:45:09.364825 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:09.364819 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8jk4z for pod openshift-network-diagnostics/network-check-target-9w6xv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:09.364967 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:09.364878 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z podName:c4ac2604-e084-4ca8-b7de-bc216ac9d843 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:25.364858736 +0000 UTC m=+33.359999487 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jk4z" (UniqueName: "kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z") pod "network-check-target-9w6xv" (UID: "c4ac2604-e084-4ca8-b7de-bc216ac9d843") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:09.539439 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:09.539362 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:09.539439 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:09.539403 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:09.539619 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:09.539471 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:09.539619 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:09.539487 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:45:09.539619 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:09.539549 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:45:09.539619 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:09.539598 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:45:11.539595 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:11.539555 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:11.540066 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:11.539606 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:11.540066 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:11.539606 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:11.540066 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:11.539691 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:45:11.540066 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:11.539772 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:45:11.540066 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:11.539859 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:45:13.539652 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:13.539620 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:13.540086 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:13.539733 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:13.540086 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:13.539740 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:45:13.540086 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:13.539747 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:13.540086 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:13.539841 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:45:13.540086 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:13.539918 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:45:14.694611 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:14.694365 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7vsds" event={"ID":"3c403d1e-9431-418e-a96a-270a9608f7ec","Type":"ContainerStarted","Data":"4d802b416eda6b964bb86c139386cd81c43a7ecba908d4505f7e0367be41e9df"} Jul 16 12:45:14.696417 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:14.696393 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-198.ec2.internal" event={"ID":"6594cb49ca46862b517c534a4f67f101","Type":"ContainerStarted","Data":"2e0089f92a1b50a7af3420023f1ba6ff8b3adf68506162ba8c633b062d8a2b43"} Jul 16 12:45:14.698310 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:14.698286 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" event={"ID":"54f5221d-4232-4b00-9970-32ae1b48e13e","Type":"ContainerStarted","Data":"cb98fbd60899ee67bd2d9bef6af2ea15e0c1dbe0418a1cd062e9adb2b99f1b94"} Jul 16 12:45:14.700750 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:14.700727 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" event={"ID":"56aed7de-c338-4c39-98d3-a6e43b58d9b7","Type":"ContainerStarted","Data":"0d3a32be1c6ef7a4053620c3bed0cef7cb6c1a836684978ce9d8dec292a970fa"} Jul 16 12:45:14.737515 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:14.737459 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wmjgp" podStartSLOduration=2.229046782 podStartE2EDuration="22.737443352s" totalImagesPullingTime="20.50839657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:14.736811843 +0000 UTC m=+22.731952596" watchObservedRunningTime="2026-07-16 12:45:14.737443352 +0000 UTC m=+22.732584112" Jul 16 12:45:14.737652 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:14.737630 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7vsds" podStartSLOduration=2.243997376 podStartE2EDuration="22.737619694s" totalImagesPullingTime="20.493622318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:14.71687103 +0000 UTC m=+22.712011791" watchObservedRunningTime="2026-07-16 12:45:14.737619694 +0000 UTC m=+22.732760445" Jul 16 12:45:14.767646 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:14.767605 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-198.ec2.internal" podStartSLOduration=21.767590973 podStartE2EDuration="21.767590973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:14.767395515 +0000 UTC m=+22.762536274" watchObservedRunningTime="2026-07-16 12:45:14.767590973 +0000 UTC m=+22.762731732" Jul 16 12:45:15.539929 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.539734 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:15.540084 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.539751 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:15.540084 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:15.540009 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:45:15.540149 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.539786 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:15.540149 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:15.540096 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:45:15.540208 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:15.540158 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:45:15.705420 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.705392 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 12:45:15.706183 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.705705 2565 generic.go:356] "Generic (PLEG): container finished" podID="56aed7de-c338-4c39-98d3-a6e43b58d9b7" containerID="734787b97dd3eafd54f3ec9af0c68d8a3a7fe5555977097fe8d826efde577d58" exitCode=1 Jul 16 12:45:15.706183 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.705781 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" event={"ID":"56aed7de-c338-4c39-98d3-a6e43b58d9b7","Type":"ContainerStarted","Data":"bba31cb17e331081e03dfe488feb7a3ed9b28d6b2745e304f1fcd1cfc58826ea"} Jul 16 12:45:15.706183 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.705870 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" event={"ID":"56aed7de-c338-4c39-98d3-a6e43b58d9b7","Type":"ContainerStarted","Data":"f220c1430903d30102c0bf088fbc949ddaadda053a96ab55ab1264afa0ed3bfd"} Jul 16 12:45:15.706183 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.705919 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" event={"ID":"56aed7de-c338-4c39-98d3-a6e43b58d9b7","Type":"ContainerStarted","Data":"9f9244529d920b1813aeb19c27ffd0a60507fc5ef763f3f42e59891389a78dcc"} Jul 16 12:45:15.706183 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.705960 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" event={"ID":"56aed7de-c338-4c39-98d3-a6e43b58d9b7","Type":"ContainerStarted","Data":"87f7028bb5495a4fce18d5098cde4fe6135609ac10d5558b5133b55588974cc3"} Jul 16 12:45:15.706183 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.706001 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" event={"ID":"56aed7de-c338-4c39-98d3-a6e43b58d9b7","Type":"ContainerDied","Data":"734787b97dd3eafd54f3ec9af0c68d8a3a7fe5555977097fe8d826efde577d58"} Jul 16 12:45:15.707145 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.707125 2565 generic.go:356] "Generic (PLEG): container finished" podID="d696b861a7ee434644a36496693ecd3e" containerID="108b993c750b8d8a24ff6e32aa22aa49f0fe4a8f05bff461b4057981bda1e15f" exitCode=0 Jul 16 12:45:15.707279 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.707190 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal" event={"ID":"d696b861a7ee434644a36496693ecd3e","Type":"ContainerDied","Data":"108b993c750b8d8a24ff6e32aa22aa49f0fe4a8f05bff461b4057981bda1e15f"} Jul 16 12:45:15.708550 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.708531 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g64cf" event={"ID":"e1549474-0299-489e-af96-767bdf28937a","Type":"ContainerStarted","Data":"2299ecdf09094e07170b830a9e15a772247cec39010eca14e85e4c4853920e4a"} Jul 16 12:45:15.710059 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.710041 2565 generic.go:356] "Generic (PLEG): container finished" podID="d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac" containerID="9dccccd16cff5ad4b3e8c1f98f9c5586715de7e2aa89483c5af5dfc5e700cc19" exitCode=0 Jul 16 12:45:15.710138 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.710104 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nv5pc" event={"ID":"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac","Type":"ContainerDied","Data":"9dccccd16cff5ad4b3e8c1f98f9c5586715de7e2aa89483c5af5dfc5e700cc19"} Jul 16 12:45:15.711791 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.711772 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v2l9k" event={"ID":"34601419-53fc-4175-b4b9-44de20f22277","Type":"ContainerStarted","Data":"a8d6e7a3491c5e6bacfb0b4bdf42ad1d3b032ebe90d9e8c031ad789bd56c599a"} Jul 16 12:45:15.713204 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.713178 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6dvg9" event={"ID":"10fa3856-33f2-4a83-9bcd-4715bdfe96b3","Type":"ContainerStarted","Data":"c85aa29fadc81da506e85ca457e23670b3074a8a1fa5ba80ed6dbb8a477863b7"} Jul 16 12:45:15.714673 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.714654 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-smgrz" event={"ID":"eb3b63d0-7a24-407c-891b-ca52cb88a27c","Type":"ContainerStarted","Data":"80813041dc52f31bc89e60da8139c59d7678554f466656baaf21a00ecd5b2b52"} Jul 16 12:45:15.716135 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.716115 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" event={"ID":"fbe4dca5-fc0f-4607-b057-73a78b901b15","Type":"ContainerStarted","Data":"60da1f4b9021080c1c6214561ac33472aa4371555c46c26d38d2cb64a303a1c0"} Jul 16 12:45:15.779802 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.779763 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6dvg9" podStartSLOduration=3.320713234 podStartE2EDuration="23.77974701s" totalImagesPullingTime="20.459033776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:15.779389634 +0000 UTC m=+23.774530394" watchObservedRunningTime="2026-07-16 12:45:15.77974701 +0000 UTC m=+23.774887769" Jul 16 12:45:15.779904 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.779878 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-smgrz" podStartSLOduration=3.234214253 podStartE2EDuration="23.779871042s" totalImagesPullingTime="20.545656789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:15.757574131 +0000 UTC m=+23.752714895" watchObservedRunningTime="2026-07-16 12:45:15.779871042 +0000 UTC m=+23.775011804" Jul 16 12:45:15.802838 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.802793 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v2l9k" podStartSLOduration=8.075408909 podStartE2EDuration="23.802775333s" totalImagesPullingTime="15.727366424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:15.802286937 +0000 UTC m=+23.797427716" watchObservedRunningTime="2026-07-16 12:45:15.802775333 +0000 UTC m=+23.797916093" Jul 16 12:45:15.824024 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:15.823985 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g64cf" podStartSLOduration=2.416776908 podStartE2EDuration="22.823970175s" totalImagesPullingTime="20.407193267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:15.823925689 +0000 UTC m=+23.819066467" watchObservedRunningTime="2026-07-16 12:45:15.823970175 +0000 UTC m=+23.819110935" Jul 16 12:45:16.434024 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:16.433983 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v2l9k" Jul 16 12:45:16.434671 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:16.434638 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v2l9k" Jul 16 12:45:16.721180 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:16.721098 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal" event={"ID":"d696b861a7ee434644a36496693ecd3e","Type":"ContainerStarted","Data":"e3d6ba8486f792b6bd061eb1e93b06ebeeb36c318a5a7a36ffa6782d3102886f"} Jul 16 12:45:16.721750 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:16.721459 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v2l9k" Jul 16 12:45:16.721750 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:16.721631 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v2l9k" Jul 16 12:45:16.742086 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:16.742035 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-198.ec2.internal" podStartSLOduration=23.74201526 podStartE2EDuration="23.74201526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:16.741287414 +0000 UTC m=+24.736428177" watchObservedRunningTime="2026-07-16 12:45:16.74201526 +0000 UTC m=+24.737156024" Jul 16 12:45:17.539514 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:17.539490 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:17.539600 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:17.539511 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:17.539664 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:17.539629 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:45:17.539946 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:17.539928 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:45:17.539989 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:17.539974 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:17.540035 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:17.540023 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:45:17.544196 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:17.544180 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 12:45:17.725575 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:17.725268 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" event={"ID":"fbe4dca5-fc0f-4607-b057-73a78b901b15","Type":"ContainerStarted","Data":"62102b02b4009d00b792d29f6b40bed3c8de00e1081fef51092382c9544ec1ef"} Jul 16 12:45:17.728992 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:17.728968 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 12:45:17.729753 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:17.729729 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" event={"ID":"56aed7de-c338-4c39-98d3-a6e43b58d9b7","Type":"ContainerStarted","Data":"9a9c25a70622b9a741c6b20a0ee1bf8928b3cc821c41c2c3d57551ac061d442f"} Jul 16 12:45:18.496804 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:18.496701 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T12:45:17.544193132Z","UUID":"4fc08396-fa84-4582-9df7-0a60970feea6","Handler":null,"Name":"","Endpoint":""} Jul 16 12:45:18.498326 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:18.498306 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 12:45:18.498447 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:18.498345 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 12:45:19.539482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:19.539448 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:19.539482 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:19.539474 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:19.539969 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:19.539474 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:19.539969 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:19.539573 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:45:19.539969 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:19.539640 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:45:19.539969 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:19.539713 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:45:20.737579 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:20.737390 2565 generic.go:356] "Generic (PLEG): container finished" podID="d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac" containerID="49b6039ff80b369df40e12c6f0b1fac15ddd6bcb5ec18d15b76d8ac8f89f1288" exitCode=0 Jul 16 12:45:20.738210 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:20.737466 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nv5pc" event={"ID":"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac","Type":"ContainerDied","Data":"49b6039ff80b369df40e12c6f0b1fac15ddd6bcb5ec18d15b76d8ac8f89f1288"} Jul 16 12:45:20.739834 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:20.739810 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" event={"ID":"fbe4dca5-fc0f-4607-b057-73a78b901b15","Type":"ContainerStarted","Data":"7b9df21f9b6884059deba0de96a59e5aa94f52c51cc7a7e912749528d2999dbe"} Jul 16 12:45:20.743049 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:20.743032 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 12:45:20.743402 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:20.743383 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" event={"ID":"56aed7de-c338-4c39-98d3-a6e43b58d9b7","Type":"ContainerStarted","Data":"3541801cb50482729cdb458ff07b9d68cc2a4575420af4e66b9eaaa4f46122bf"} Jul 16 12:45:20.743671 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:20.743634 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:45:20.743810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:20.743715 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:45:20.743810 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:20.743760 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:45:20.743912 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:20.743814 2565 scope.go:117] "RemoveContainer" containerID="734787b97dd3eafd54f3ec9af0c68d8a3a7fe5555977097fe8d826efde577d58" Jul 16 12:45:20.760574 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:20.760555 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:45:20.760907 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:20.760888 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:45:20.834553 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:20.834451 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqr28" podStartSLOduration=2.316512206 podStartE2EDuration="27.834436314s" totalImagesPullingTime="25.517924108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:44:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:20.834153268 +0000 UTC m=+28.829294029" watchObservedRunningTime="2026-07-16 12:45:20.834436314 +0000 UTC m=+28.829577073" Jul 16 12:45:21.539828 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:21.539735 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:21.539828 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:21.539781 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:21.539828 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:21.539788 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:21.540064 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:21.539863 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:45:21.540064 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:21.539991 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:45:21.540155 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:21.540093 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:45:21.749721 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:21.749702 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 12:45:21.750120 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:21.750088 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" event={"ID":"56aed7de-c338-4c39-98d3-a6e43b58d9b7","Type":"ContainerStarted","Data":"8d8a1b012fa42538bf3bd458abb27edb461e05ad7731dc1205a4530fb95e6e93"} Jul 16 12:45:21.789127 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:21.789083 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" podStartSLOduration=9.050031639 podStartE2EDuration="29.789069673s" totalImagesPullingTime="20.739038034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:21.788819342 +0000 UTC m=+29.783960101" watchObservedRunningTime="2026-07-16 12:45:21.789069673 +0000 UTC m=+29.784210432" Jul 16 12:45:22.553537 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:22.553510 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zdrc5"] Jul 16 12:45:22.553719 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:22.553614 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:22.553781 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:22.553724 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:45:22.557169 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:22.557141 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9w6xv"] Jul 16 12:45:22.557306 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:22.557250 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:22.557373 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:22.557352 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:45:22.557698 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:22.557679 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n2c94"] Jul 16 12:45:22.557804 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:22.557748 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:22.557863 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:22.557828 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:45:22.754440 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:22.754403 2565 generic.go:356] "Generic (PLEG): container finished" podID="d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac" containerID="aa36ae9e1688596c494b0edcc51094f401ec53a343170e1e2d88c72b8bb47368" exitCode=0 Jul 16 12:45:22.754911 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:22.754481 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nv5pc" event={"ID":"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac","Type":"ContainerDied","Data":"aa36ae9e1688596c494b0edcc51094f401ec53a343170e1e2d88c72b8bb47368"} Jul 16 12:45:24.539387 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:24.539123 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:24.539852 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:24.539126 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:24.539852 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:24.539428 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:45:24.539852 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:24.539176 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:24.539852 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:24.539528 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:45:24.539852 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:24.539584 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:45:24.761192 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:24.761160 2565 generic.go:356] "Generic (PLEG): container finished" podID="d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac" containerID="7b0df83b3edecbd1b68f0de3a1a117121c6c8adfac7150ccdbc8e9e5e976cbbf" exitCode=0 Jul 16 12:45:24.761331 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:24.761215 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nv5pc" event={"ID":"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac","Type":"ContainerDied","Data":"7b0df83b3edecbd1b68f0de3a1a117121c6c8adfac7150ccdbc8e9e5e976cbbf"} Jul 16 12:45:25.182591 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:25.182548 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs\") pod \"network-metrics-daemon-n2c94\" (UID: \"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe\") " pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:25.182767 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:25.182716 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:25.182828 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:25.182796 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs podName:dd3f2e5f-9499-4947-8d80-160b3ffd0bbe nodeName:}" failed. No retries permitted until 2026-07-16 12:45:57.182773862 +0000 UTC m=+65.177914619 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs") pod "network-metrics-daemon-n2c94" (UID: "dd3f2e5f-9499-4947-8d80-160b3ffd0bbe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:25.284023 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:25.283981 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret\") pod \"global-pull-secret-syncer-zdrc5\" (UID: \"5de8db0d-cb13-482f-b0f6-8259d3cb37f9\") " pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:25.284166 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:25.284125 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:25.284209 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:25.284180 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret podName:5de8db0d-cb13-482f-b0f6-8259d3cb37f9 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:57.284166639 +0000 UTC m=+65.279307377 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret") pod "global-pull-secret-syncer-zdrc5" (UID: "5de8db0d-cb13-482f-b0f6-8259d3cb37f9") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:25.385286 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:25.385245 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jk4z\" (UniqueName: \"kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z\") pod \"network-check-target-9w6xv\" (UID: \"c4ac2604-e084-4ca8-b7de-bc216ac9d843\") " pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:25.385439 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:25.385409 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:45:25.385439 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:25.385435 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:45:25.385525 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:25.385449 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8jk4z for pod openshift-network-diagnostics/network-check-target-9w6xv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:25.385525 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:25.385516 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z podName:c4ac2604-e084-4ca8-b7de-bc216ac9d843 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:57.385498533 +0000 UTC m=+65.380639284 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jk4z" (UniqueName: "kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z") pod "network-check-target-9w6xv" (UID: "c4ac2604-e084-4ca8-b7de-bc216ac9d843") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:26.539128 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:26.539092 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:26.539705 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:26.539194 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:26.539705 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:26.539209 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9w6xv" podUID="c4ac2604-e084-4ca8-b7de-bc216ac9d843" Jul 16 12:45:26.539705 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:26.539315 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdrc5" podUID="5de8db0d-cb13-482f-b0f6-8259d3cb37f9" Jul 16 12:45:26.539705 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:26.539348 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:26.539705 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:26.539429 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2c94" podUID="dd3f2e5f-9499-4947-8d80-160b3ffd0bbe" Jul 16 12:45:27.251972 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.251754 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-198.ec2.internal" event="NodeReady" Jul 16 12:45:27.251972 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.251903 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 12:45:27.309634 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.309595 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55c4f54f86-chtl8"] Jul 16 12:45:27.328115 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.328089 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5dxp9"] Jul 16 12:45:27.328280 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.328245 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.331171 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.331145 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 12:45:27.331317 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.331225 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 12:45:27.331468 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.331447 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 12:45:27.331570 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.331542 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rr49k\"" Jul 16 12:45:27.337142 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.337123 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6n5wk"] Jul 16 12:45:27.337700 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.337680 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 12:45:27.338302 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.338282 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:27.340956 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.340712 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 12:45:27.340956 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.340713 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 12:45:27.340956 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.340787 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lnsbk\"" Jul 16 12:45:27.350063 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.349977 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55c4f54f86-chtl8"] Jul 16 12:45:27.350063 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.350021 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5dxp9"] Jul 16 12:45:27.350063 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.350042 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6n5wk"] Jul 16 12:45:27.350283 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.350079 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6n5wk" Jul 16 12:45:27.352678 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.352659 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 12:45:27.352878 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.352675 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 12:45:27.353008 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.352675 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wlnmp\"" Jul 16 12:45:27.353008 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.352748 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 12:45:27.502264 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.502173 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd384a9f-61e7-4509-8346-5d345021a603-tmp-dir\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:27.502264 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.502219 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnbgx\" (UniqueName: \"kubernetes.io/projected/bd384a9f-61e7-4509-8346-5d345021a603-kube-api-access-mnbgx\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:27.502484 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.502361 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.502484 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.502394 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9eb17f11-3a02-4674-b023-7b781b393e38-ca-trust-extracted\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.502484 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.502412 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-bound-sa-token\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.502484 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.502436 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd384a9f-61e7-4509-8346-5d345021a603-config-volume\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:27.502484 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.502456 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:27.502700 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.502486 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9eb17f11-3a02-4674-b023-7b781b393e38-image-registry-private-configuration\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.502700 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.502515 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9eb17f11-3a02-4674-b023-7b781b393e38-registry-certificates\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.502700 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.502537 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q9p4\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-kube-api-access-9q9p4\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.502700 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.502558 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/427c004b-ed80-4024-88ab-b817224073b4-cert\") pod \"ingress-canary-6n5wk\" (UID: \"427c004b-ed80-4024-88ab-b817224073b4\") " pod="openshift-ingress-canary/ingress-canary-6n5wk" Jul 16 12:45:27.502700 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.502587 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9eb17f11-3a02-4674-b023-7b781b393e38-installation-pull-secrets\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.502700 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.502612 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5pbx\" (UniqueName: \"kubernetes.io/projected/427c004b-ed80-4024-88ab-b817224073b4-kube-api-access-c5pbx\") pod \"ingress-canary-6n5wk\" (UID: \"427c004b-ed80-4024-88ab-b817224073b4\") " pod="openshift-ingress-canary/ingress-canary-6n5wk" Jul 16 12:45:27.502700 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.502649 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9eb17f11-3a02-4674-b023-7b781b393e38-trusted-ca\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.508800 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.508769 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v2l9k_34601419-53fc-4175-b4b9-44de20f22277/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:27.603854 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.603822 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.603854 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.603858 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9eb17f11-3a02-4674-b023-7b781b393e38-ca-trust-extracted\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.603878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-bound-sa-token\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:27.603978 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:27.603992 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55c4f54f86-chtl8: secret "image-registry-tls" not found Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.604025 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd384a9f-61e7-4509-8346-5d345021a603-config-volume\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:27.604044 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls podName:9eb17f11-3a02-4674-b023-7b781b393e38 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:28.104025371 +0000 UTC m=+36.099166109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls") pod "image-registry-55c4f54f86-chtl8" (UID: "9eb17f11-3a02-4674-b023-7b781b393e38") : secret "image-registry-tls" not found Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.604068 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.604092 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9eb17f11-3a02-4674-b023-7b781b393e38-image-registry-private-configuration\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.604122 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9eb17f11-3a02-4674-b023-7b781b393e38-registry-certificates\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.604144 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q9p4\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-kube-api-access-9q9p4\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:27.604164 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.604193 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/427c004b-ed80-4024-88ab-b817224073b4-cert\") pod \"ingress-canary-6n5wk\" (UID: \"427c004b-ed80-4024-88ab-b817224073b4\") " pod="openshift-ingress-canary/ingress-canary-6n5wk" Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:27.604223 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls podName:bd384a9f-61e7-4509-8346-5d345021a603 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:28.10420534 +0000 UTC m=+36.099346078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls") pod "dns-default-5dxp9" (UID: "bd384a9f-61e7-4509-8346-5d345021a603") : secret "dns-default-metrics-tls" not found Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.604274 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9eb17f11-3a02-4674-b023-7b781b393e38-installation-pull-secrets\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.604297 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5pbx\" (UniqueName: \"kubernetes.io/projected/427c004b-ed80-4024-88ab-b817224073b4-kube-api-access-c5pbx\") pod \"ingress-canary-6n5wk\" (UID: \"427c004b-ed80-4024-88ab-b817224073b4\") " pod="openshift-ingress-canary/ingress-canary-6n5wk" Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.604327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9eb17f11-3a02-4674-b023-7b781b393e38-trusted-ca\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.604387 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd384a9f-61e7-4509-8346-5d345021a603-tmp-dir\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:27.604538 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.604409 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbgx\" (UniqueName: \"kubernetes.io/projected/bd384a9f-61e7-4509-8346-5d345021a603-kube-api-access-mnbgx\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:27.605297 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.604730 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd384a9f-61e7-4509-8346-5d345021a603-tmp-dir\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:27.605297 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.604831 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9eb17f11-3a02-4674-b023-7b781b393e38-ca-trust-extracted\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.605297 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.604847 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd384a9f-61e7-4509-8346-5d345021a603-config-volume\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:27.605297 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.604935 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9eb17f11-3a02-4674-b023-7b781b393e38-registry-certificates\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.605495 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.605298 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9eb17f11-3a02-4674-b023-7b781b393e38-trusted-ca\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.608679 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.608656 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9eb17f11-3a02-4674-b023-7b781b393e38-image-registry-private-configuration\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.608679 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.608669 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9eb17f11-3a02-4674-b023-7b781b393e38-installation-pull-secrets\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.608846 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.608661 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/427c004b-ed80-4024-88ab-b817224073b4-cert\") pod \"ingress-canary-6n5wk\" (UID: \"427c004b-ed80-4024-88ab-b817224073b4\") " pod="openshift-ingress-canary/ingress-canary-6n5wk" Jul 16 12:45:27.614438 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.614410 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q9p4\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-kube-api-access-9q9p4\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.614914 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.614891 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-bound-sa-token\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:27.618822 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.618801 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnbgx\" (UniqueName: \"kubernetes.io/projected/bd384a9f-61e7-4509-8346-5d345021a603-kube-api-access-mnbgx\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:27.620114 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.620087 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5pbx\" (UniqueName: \"kubernetes.io/projected/427c004b-ed80-4024-88ab-b817224073b4-kube-api-access-c5pbx\") pod \"ingress-canary-6n5wk\" (UID: \"427c004b-ed80-4024-88ab-b817224073b4\") " pod="openshift-ingress-canary/ingress-canary-6n5wk" Jul 16 12:45:27.638847 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.638817 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v2l9k_34601419-53fc-4175-b4b9-44de20f22277/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:27.658933 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:27.658905 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6n5wk" Jul 16 12:45:28.109671 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:28.109621 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:28.109855 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:28.109681 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:28.109855 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:28.109788 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:45:28.109855 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:28.109794 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:45:28.109855 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:28.109818 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55c4f54f86-chtl8: secret "image-registry-tls" not found Jul 16 12:45:28.110022 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:28.109862 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls podName:bd384a9f-61e7-4509-8346-5d345021a603 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:29.109843799 +0000 UTC m=+37.104984537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls") pod "dns-default-5dxp9" (UID: "bd384a9f-61e7-4509-8346-5d345021a603") : secret "dns-default-metrics-tls" not found Jul 16 12:45:28.110022 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:28.109883 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls podName:9eb17f11-3a02-4674-b023-7b781b393e38 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:29.109869188 +0000 UTC m=+37.105009956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls") pod "image-registry-55c4f54f86-chtl8" (UID: "9eb17f11-3a02-4674-b023-7b781b393e38") : secret "image-registry-tls" not found Jul 16 12:45:28.539828 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:28.539792 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:28.540014 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:28.539793 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:28.540014 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:28.539965 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:28.542565 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:28.542540 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 12:45:28.543599 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:28.543225 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 12:45:28.543599 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:28.543319 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sbdj2\"" Jul 16 12:45:28.543599 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:28.543360 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 12:45:28.543599 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:28.543498 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xrlr5\"" Jul 16 12:45:28.543599 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:28.543513 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 12:45:28.959842 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:28.959649 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6n5wk"] Jul 16 12:45:29.040401 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:45:29.040371 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427c004b_ed80_4024_88ab_b817224073b4.slice/crio-7ccddcc372d666355ec7fdc3770fcbf66f55c943379b1fe8ab01cb569dd965be WatchSource:0}: Error finding container 7ccddcc372d666355ec7fdc3770fcbf66f55c943379b1fe8ab01cb569dd965be: Status 404 returned error can't find the container with id 7ccddcc372d666355ec7fdc3770fcbf66f55c943379b1fe8ab01cb569dd965be Jul 16 12:45:29.118189 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:29.118164 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:29.118297 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:29.118194 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:29.118340 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:29.118315 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:45:29.118340 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:29.118320 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:45:29.118340 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:29.118334 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55c4f54f86-chtl8: secret "image-registry-tls" not found Jul 16 12:45:29.118427 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:29.118372 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls podName:bd384a9f-61e7-4509-8346-5d345021a603 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:31.11835961 +0000 UTC m=+39.113500347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls") pod "dns-default-5dxp9" (UID: "bd384a9f-61e7-4509-8346-5d345021a603") : secret "dns-default-metrics-tls" not found Jul 16 12:45:29.118427 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:29.118390 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls podName:9eb17f11-3a02-4674-b023-7b781b393e38 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:31.118376867 +0000 UTC m=+39.113517617 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls") pod "image-registry-55c4f54f86-chtl8" (UID: "9eb17f11-3a02-4674-b023-7b781b393e38") : secret "image-registry-tls" not found Jul 16 12:45:29.317137 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:29.317079 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v2l9k_34601419-53fc-4175-b4b9-44de20f22277/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:29.402447 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:29.402420 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v2l9k_34601419-53fc-4175-b4b9-44de20f22277/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:29.774415 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:29.774383 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6n5wk" event={"ID":"427c004b-ed80-4024-88ab-b817224073b4","Type":"ContainerStarted","Data":"7ccddcc372d666355ec7fdc3770fcbf66f55c943379b1fe8ab01cb569dd965be"} Jul 16 12:45:29.776706 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:29.776683 2565 generic.go:356] "Generic (PLEG): container finished" podID="d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac" containerID="2e144e9b96e0067bcd10f5922778a563c3d066a0c11354e3b556e489e091cfe6" exitCode=0 Jul 16 12:45:29.776827 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:29.776723 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nv5pc" event={"ID":"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac","Type":"ContainerDied","Data":"2e144e9b96e0067bcd10f5922778a563c3d066a0c11354e3b556e489e091cfe6"} Jul 16 12:45:30.782198 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:30.782156 2565 generic.go:356] "Generic (PLEG): container finished" podID="d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac" containerID="ec5a8cecd120dc0a3ce6261f24d1408466000caf394eec3e48cb71f57ae55daa" exitCode=0 Jul 16 12:45:30.782727 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:30.782250 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nv5pc" event={"ID":"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac","Type":"ContainerDied","Data":"ec5a8cecd120dc0a3ce6261f24d1408466000caf394eec3e48cb71f57ae55daa"} Jul 16 12:45:31.133581 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:31.133489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:31.133581 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:31.133541 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:31.133796 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:31.133665 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:45:31.133796 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:31.133688 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55c4f54f86-chtl8: secret "image-registry-tls" not found Jul 16 12:45:31.133796 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:31.133704 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:45:31.133796 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:31.133747 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls podName:9eb17f11-3a02-4674-b023-7b781b393e38 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:35.133729287 +0000 UTC m=+43.128870030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls") pod "image-registry-55c4f54f86-chtl8" (UID: "9eb17f11-3a02-4674-b023-7b781b393e38") : secret "image-registry-tls" not found Jul 16 12:45:31.133796 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:31.133762 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls podName:bd384a9f-61e7-4509-8346-5d345021a603 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:35.133755897 +0000 UTC m=+43.128896634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls") pod "dns-default-5dxp9" (UID: "bd384a9f-61e7-4509-8346-5d345021a603") : secret "dns-default-metrics-tls" not found Jul 16 12:45:31.788787 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:31.788746 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nv5pc" event={"ID":"d08b5ee1-e4dc-4f6c-b2a9-d8b6d560eaac","Type":"ContainerStarted","Data":"d3e90f4975d84419c63a474c898b1a27c4ae5cde570851cb1777bdb117aa77d5"} Jul 16 12:45:31.817581 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:31.817530 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nv5pc" podStartSLOduration=8.510271527 podStartE2EDuration="39.817509111s" totalImagesPullingTime="31.307237584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:44:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:31.816947248 +0000 UTC m=+39.812088021" watchObservedRunningTime="2026-07-16 12:45:31.817509111 +0000 UTC m=+39.812649905" Jul 16 12:45:32.073254 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:32.072959 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v2l9k_34601419-53fc-4175-b4b9-44de20f22277/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:32.181850 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:32.181814 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v2l9k_34601419-53fc-4175-b4b9-44de20f22277/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:33.794646 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:33.794609 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6n5wk" event={"ID":"427c004b-ed80-4024-88ab-b817224073b4","Type":"ContainerStarted","Data":"c0b7337ea50f5f5a5d44b579cec304f25b2ac679a5a3f4604d10970fc77791d6"} Jul 16 12:45:33.817924 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:33.817881 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6n5wk" podStartSLOduration=3.118113076 podStartE2EDuration="6.81786696s" totalImagesPullingTime="3.699753884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:33.816384791 +0000 UTC m=+41.811525550" watchObservedRunningTime="2026-07-16 12:45:33.81786696 +0000 UTC m=+41.813007720" Jul 16 12:45:35.164366 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:35.164322 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:35.164366 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:35.164367 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:35.164773 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:35.164480 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:45:35.164773 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:35.164490 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:45:35.164773 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:35.164508 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55c4f54f86-chtl8: secret "image-registry-tls" not found Jul 16 12:45:35.164773 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:35.164532 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls podName:bd384a9f-61e7-4509-8346-5d345021a603 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:43.164518457 +0000 UTC m=+51.159659195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls") pod "dns-default-5dxp9" (UID: "bd384a9f-61e7-4509-8346-5d345021a603") : secret "dns-default-metrics-tls" not found Jul 16 12:45:35.164773 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:35.164551 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls podName:9eb17f11-3a02-4674-b023-7b781b393e38 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:43.164538191 +0000 UTC m=+51.159678928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls") pod "image-registry-55c4f54f86-chtl8" (UID: "9eb17f11-3a02-4674-b023-7b781b393e38") : secret "image-registry-tls" not found Jul 16 12:45:36.169653 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:36.169607 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v2l9k_34601419-53fc-4175-b4b9-44de20f22277/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:36.277610 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:36.277583 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v2l9k_34601419-53fc-4175-b4b9-44de20f22277/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:43.223886 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:43.223849 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:43.223886 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:43.223888 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:43.224386 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:43.224023 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:45:43.224386 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:43.224025 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:45:43.224386 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:43.224052 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55c4f54f86-chtl8: secret "image-registry-tls" not found Jul 16 12:45:43.224386 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:43.224084 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls podName:bd384a9f-61e7-4509-8346-5d345021a603 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:59.224069269 +0000 UTC m=+67.219210020 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls") pod "dns-default-5dxp9" (UID: "bd384a9f-61e7-4509-8346-5d345021a603") : secret "dns-default-metrics-tls" not found Jul 16 12:45:43.224386 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:43.224120 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls podName:9eb17f11-3a02-4674-b023-7b781b393e38 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:59.224101609 +0000 UTC m=+67.219242362 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls") pod "image-registry-55c4f54f86-chtl8" (UID: "9eb17f11-3a02-4674-b023-7b781b393e38") : secret "image-registry-tls" not found Jul 16 12:45:50.267078 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.267040 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-qbhg8"] Jul 16 12:45:50.268992 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.268976 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-75f7d966c4-bb2lf"] Jul 16 12:45:50.269124 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.269107 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.270782 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.270762 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:50.271924 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.271904 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 16 12:45:50.272365 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.272352 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 12:45:50.273031 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.272993 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 12:45:50.273031 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.273000 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 16 12:45:50.273263 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.273096 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 16 12:45:50.273263 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.273135 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 16 12:45:50.273263 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.273149 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zc5m4\"" Jul 16 12:45:50.273466 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.273274 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jmjpw\"" Jul 16 12:45:50.273466 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.273302 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 16 12:45:50.273569 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.273512 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 16 12:45:50.273569 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.273538 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 12:45:50.274159 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.274138 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 12:45:50.277923 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.277901 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 16 12:45:50.280747 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.280726 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-qbhg8"] Jul 16 12:45:50.281393 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.281373 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75f7d966c4-bb2lf"] Jul 16 12:45:50.360268 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.360222 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr"] Jul 16 12:45:50.362026 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.361996 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" Jul 16 12:45:50.364179 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.364150 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc"] Jul 16 12:45:50.364327 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.364307 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 16 12:45:50.364468 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.364402 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 16 12:45:50.364562 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.364517 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:45:50.365042 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.365027 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h4nj6\"" Jul 16 12:45:50.365821 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.365805 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7"] Jul 16 12:45:50.365928 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.365915 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc" Jul 16 12:45:50.367477 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.367458 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:45:50.368809 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.368790 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 16 12:45:50.369048 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.369022 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:45:50.369191 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.369175 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 16 12:45:50.369312 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.369247 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7d7xh\"" Jul 16 12:45:50.369991 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.369972 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 12:45:50.370112 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.370085 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 12:45:50.370112 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.370093 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 16 12:45:50.370246 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.370181 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 16 12:45:50.370336 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.370318 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 16 12:45:50.370585 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.370571 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6kbfr\"" Jul 16 12:45:50.375477 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.375433 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-default-certificate\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:50.375477 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.375467 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73045a2a-1f69-48c1-96f1-54b941192680-serving-cert\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.375653 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.375513 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73045a2a-1f69-48c1-96f1-54b941192680-tmp\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.375653 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.375552 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:50.375653 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.375575 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:50.375653 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.375595 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kfp5\" (UniqueName: \"kubernetes.io/projected/15790d2d-298f-4fff-9945-c8bc693a04b1-kube-api-access-5kfp5\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:50.375653 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.375617 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73045a2a-1f69-48c1-96f1-54b941192680-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.375832 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.375658 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mqwlr\" (UID: \"e3c7c7d3-f2da-4987-aadc-5cc832b0c234\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" Jul 16 12:45:50.375832 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.375681 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-stats-auth\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:50.375832 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.375713 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5grx\" (UniqueName: \"kubernetes.io/projected/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-kube-api-access-s5grx\") pod \"cluster-samples-operator-65db858894-mqwlr\" (UID: \"e3c7c7d3-f2da-4987-aadc-5cc832b0c234\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" Jul 16 12:45:50.375832 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.375750 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/73045a2a-1f69-48c1-96f1-54b941192680-snapshots\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.375832 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.375813 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6q2h\" (UniqueName: \"kubernetes.io/projected/5e27e160-a28d-4523-a137-fb415f779ba5-kube-api-access-n6q2h\") pod \"service-ca-operator-7695fc6c8c-brblc\" (UID: \"5e27e160-a28d-4523-a137-fb415f779ba5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc" Jul 16 12:45:50.375994 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.375878 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e27e160-a28d-4523-a137-fb415f779ba5-config\") pod \"service-ca-operator-7695fc6c8c-brblc\" (UID: \"5e27e160-a28d-4523-a137-fb415f779ba5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc" Jul 16 12:45:50.375994 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.375902 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e27e160-a28d-4523-a137-fb415f779ba5-serving-cert\") pod \"service-ca-operator-7695fc6c8c-brblc\" (UID: \"5e27e160-a28d-4523-a137-fb415f779ba5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc" Jul 16 12:45:50.375994 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.375927 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73045a2a-1f69-48c1-96f1-54b941192680-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.375994 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.375949 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll4sd\" (UniqueName: \"kubernetes.io/projected/73045a2a-1f69-48c1-96f1-54b941192680-kube-api-access-ll4sd\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.383256 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.380216 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr"] Jul 16 12:45:50.385542 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.385518 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc"] Jul 16 12:45:50.386805 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.386785 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7"] Jul 16 12:45:50.477022 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.476983 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:50.477022 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477020 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:50.477286 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477048 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kfp5\" (UniqueName: \"kubernetes.io/projected/15790d2d-298f-4fff-9945-c8bc693a04b1-kube-api-access-5kfp5\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:50.477286 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477073 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73045a2a-1f69-48c1-96f1-54b941192680-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.477286 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mqwlr\" (UID: \"e3c7c7d3-f2da-4987-aadc-5cc832b0c234\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" Jul 16 12:45:50.477286 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477138 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-stats-auth\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:50.477286 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:50.477151 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 12:45:50.477286 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477167 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5grx\" (UniqueName: \"kubernetes.io/projected/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-kube-api-access-s5grx\") pod \"cluster-samples-operator-65db858894-mqwlr\" (UID: \"e3c7c7d3-f2da-4987-aadc-5cc832b0c234\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" Jul 16 12:45:50.477286 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:50.477184 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle podName:15790d2d-298f-4fff-9945-c8bc693a04b1 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:50.977162955 +0000 UTC m=+58.972303713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle") pod "router-default-75f7d966c4-bb2lf" (UID: "15790d2d-298f-4fff-9945-c8bc693a04b1") : configmap references non-existent config key: service-ca.crt Jul 16 12:45:50.477286 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:50.477216 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs podName:15790d2d-298f-4fff-9945-c8bc693a04b1 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:50.977198284 +0000 UTC m=+58.972339033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs") pod "router-default-75f7d966c4-bb2lf" (UID: "15790d2d-298f-4fff-9945-c8bc693a04b1") : secret "router-metrics-certs-default" not found Jul 16 12:45:50.477286 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477282 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/73045a2a-1f69-48c1-96f1-54b941192680-snapshots\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.477723 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477312 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6q2h\" (UniqueName: \"kubernetes.io/projected/5e27e160-a28d-4523-a137-fb415f779ba5-kube-api-access-n6q2h\") pod \"service-ca-operator-7695fc6c8c-brblc\" (UID: \"5e27e160-a28d-4523-a137-fb415f779ba5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc" Jul 16 12:45:50.477723 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:50.477325 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 12:45:50.477723 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477350 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c7ng7\" (UID: \"7a6c1393-8665-4d4a-9238-77f24e498e50\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:45:50.477723 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:50.477388 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls podName:e3c7c7d3-f2da-4987-aadc-5cc832b0c234 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:50.977372315 +0000 UTC m=+58.972513056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls") pod "cluster-samples-operator-65db858894-mqwlr" (UID: "e3c7c7d3-f2da-4987-aadc-5cc832b0c234") : secret "samples-operator-tls" not found Jul 16 12:45:50.477723 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477413 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgcrt\" (UniqueName: \"kubernetes.io/projected/7a6c1393-8665-4d4a-9238-77f24e498e50-kube-api-access-dgcrt\") pod \"cluster-monitoring-operator-5f7f494dd4-c7ng7\" (UID: \"7a6c1393-8665-4d4a-9238-77f24e498e50\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:45:50.477723 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477474 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e27e160-a28d-4523-a137-fb415f779ba5-config\") pod \"service-ca-operator-7695fc6c8c-brblc\" (UID: \"5e27e160-a28d-4523-a137-fb415f779ba5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc" Jul 16 12:45:50.477723 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477507 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e27e160-a28d-4523-a137-fb415f779ba5-serving-cert\") pod \"service-ca-operator-7695fc6c8c-brblc\" (UID: \"5e27e160-a28d-4523-a137-fb415f779ba5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc" Jul 16 12:45:50.477723 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477534 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73045a2a-1f69-48c1-96f1-54b941192680-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.477723 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477561 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll4sd\" (UniqueName: \"kubernetes.io/projected/73045a2a-1f69-48c1-96f1-54b941192680-kube-api-access-ll4sd\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.477723 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477594 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-default-certificate\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:50.477723 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477620 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73045a2a-1f69-48c1-96f1-54b941192680-serving-cert\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.477723 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73045a2a-1f69-48c1-96f1-54b941192680-tmp\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.477723 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477703 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7a6c1393-8665-4d4a-9238-77f24e498e50-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-c7ng7\" (UID: \"7a6c1393-8665-4d4a-9238-77f24e498e50\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:45:50.478271 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477877 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73045a2a-1f69-48c1-96f1-54b941192680-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.478271 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.477982 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/73045a2a-1f69-48c1-96f1-54b941192680-snapshots\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.478271 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.478147 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73045a2a-1f69-48c1-96f1-54b941192680-tmp\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.478271 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.478113 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e27e160-a28d-4523-a137-fb415f779ba5-config\") pod \"service-ca-operator-7695fc6c8c-brblc\" (UID: \"5e27e160-a28d-4523-a137-fb415f779ba5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc" Jul 16 12:45:50.478454 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.478335 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73045a2a-1f69-48c1-96f1-54b941192680-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.481174 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.481150 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e27e160-a28d-4523-a137-fb415f779ba5-serving-cert\") pod \"service-ca-operator-7695fc6c8c-brblc\" (UID: \"5e27e160-a28d-4523-a137-fb415f779ba5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc" Jul 16 12:45:50.481174 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.481161 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73045a2a-1f69-48c1-96f1-54b941192680-serving-cert\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.481339 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.481197 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-stats-auth\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:50.481414 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.481394 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-default-certificate\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:50.495568 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.495538 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kfp5\" (UniqueName: \"kubernetes.io/projected/15790d2d-298f-4fff-9945-c8bc693a04b1-kube-api-access-5kfp5\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:50.495915 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.495895 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5grx\" (UniqueName: \"kubernetes.io/projected/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-kube-api-access-s5grx\") pod \"cluster-samples-operator-65db858894-mqwlr\" (UID: \"e3c7c7d3-f2da-4987-aadc-5cc832b0c234\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" Jul 16 12:45:50.499133 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.499111 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll4sd\" (UniqueName: \"kubernetes.io/projected/73045a2a-1f69-48c1-96f1-54b941192680-kube-api-access-ll4sd\") pod \"insights-operator-68d9dd94cf-qbhg8\" (UID: \"73045a2a-1f69-48c1-96f1-54b941192680\") " pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.501623 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.501598 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6q2h\" (UniqueName: \"kubernetes.io/projected/5e27e160-a28d-4523-a137-fb415f779ba5-kube-api-access-n6q2h\") pod \"service-ca-operator-7695fc6c8c-brblc\" (UID: \"5e27e160-a28d-4523-a137-fb415f779ba5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc" Jul 16 12:45:50.578516 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.578425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c7ng7\" (UID: \"7a6c1393-8665-4d4a-9238-77f24e498e50\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:45:50.578516 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.578464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgcrt\" (UniqueName: \"kubernetes.io/projected/7a6c1393-8665-4d4a-9238-77f24e498e50-kube-api-access-dgcrt\") pod \"cluster-monitoring-operator-5f7f494dd4-c7ng7\" (UID: \"7a6c1393-8665-4d4a-9238-77f24e498e50\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:45:50.578688 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.578539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7a6c1393-8665-4d4a-9238-77f24e498e50-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-c7ng7\" (UID: \"7a6c1393-8665-4d4a-9238-77f24e498e50\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:45:50.578688 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:50.578589 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 12:45:50.578688 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:50.578674 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls podName:7a6c1393-8665-4d4a-9238-77f24e498e50 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:51.078654681 +0000 UTC m=+59.073795441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-c7ng7" (UID: "7a6c1393-8665-4d4a-9238-77f24e498e50") : secret "cluster-monitoring-operator-tls" not found Jul 16 12:45:50.579219 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.579199 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7a6c1393-8665-4d4a-9238-77f24e498e50-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-c7ng7\" (UID: \"7a6c1393-8665-4d4a-9238-77f24e498e50\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:45:50.581335 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.581320 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" Jul 16 12:45:50.595528 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.595506 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgcrt\" (UniqueName: \"kubernetes.io/projected/7a6c1393-8665-4d4a-9238-77f24e498e50-kube-api-access-dgcrt\") pod \"cluster-monitoring-operator-5f7f494dd4-c7ng7\" (UID: \"7a6c1393-8665-4d4a-9238-77f24e498e50\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:45:50.677780 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.677750 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc" Jul 16 12:45:50.711084 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.711039 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-qbhg8"] Jul 16 12:45:50.812362 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.812331 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc"] Jul 16 12:45:50.815685 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:45:50.815659 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e27e160_a28d_4523_a137_fb415f779ba5.slice/crio-ef18b4d10d47447ee104734c4f7eb25b00dada7ebb788e189c1e72975516a951 WatchSource:0}: Error finding container ef18b4d10d47447ee104734c4f7eb25b00dada7ebb788e189c1e72975516a951: Status 404 returned error can't find the container with id ef18b4d10d47447ee104734c4f7eb25b00dada7ebb788e189c1e72975516a951 Jul 16 12:45:50.831731 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.831664 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc" event={"ID":"5e27e160-a28d-4523-a137-fb415f779ba5","Type":"ContainerStarted","Data":"ef18b4d10d47447ee104734c4f7eb25b00dada7ebb788e189c1e72975516a951"} Jul 16 12:45:50.832664 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.832640 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" event={"ID":"73045a2a-1f69-48c1-96f1-54b941192680","Type":"ContainerStarted","Data":"f8e1d4764e3c61ba945feea9c8502b72890f832ff2a5268d863c515dc3829f01"} Jul 16 12:45:50.980974 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.980932 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:50.980974 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.980975 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:50.981195 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:50.981003 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mqwlr\" (UID: \"e3c7c7d3-f2da-4987-aadc-5cc832b0c234\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" Jul 16 12:45:50.981195 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:50.981091 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 12:45:50.981195 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:50.981091 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 12:45:50.981195 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:50.981105 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle podName:15790d2d-298f-4fff-9945-c8bc693a04b1 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:51.981089201 +0000 UTC m=+59.976229938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle") pod "router-default-75f7d966c4-bb2lf" (UID: "15790d2d-298f-4fff-9945-c8bc693a04b1") : configmap references non-existent config key: service-ca.crt Jul 16 12:45:50.981377 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:50.981270 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls podName:e3c7c7d3-f2da-4987-aadc-5cc832b0c234 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:51.981253003 +0000 UTC m=+59.976393741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls") pod "cluster-samples-operator-65db858894-mqwlr" (UID: "e3c7c7d3-f2da-4987-aadc-5cc832b0c234") : secret "samples-operator-tls" not found Jul 16 12:45:50.981377 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:50.981295 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs podName:15790d2d-298f-4fff-9945-c8bc693a04b1 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:51.98128869 +0000 UTC m=+59.976429428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs") pod "router-default-75f7d966c4-bb2lf" (UID: "15790d2d-298f-4fff-9945-c8bc693a04b1") : secret "router-metrics-certs-default" not found Jul 16 12:45:51.081653 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:51.081597 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c7ng7\" (UID: \"7a6c1393-8665-4d4a-9238-77f24e498e50\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:45:51.081876 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:51.081748 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 12:45:51.081876 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:51.081813 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls podName:7a6c1393-8665-4d4a-9238-77f24e498e50 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:52.081798825 +0000 UTC m=+60.076939566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-c7ng7" (UID: "7a6c1393-8665-4d4a-9238-77f24e498e50") : secret "cluster-monitoring-operator-tls" not found Jul 16 12:45:51.991656 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:51.990732 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:51.991656 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:51.990788 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:51.991656 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:51.990840 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mqwlr\" (UID: \"e3c7c7d3-f2da-4987-aadc-5cc832b0c234\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" Jul 16 12:45:51.991656 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:51.991040 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 12:45:51.991656 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:51.991128 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls podName:e3c7c7d3-f2da-4987-aadc-5cc832b0c234 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:53.991114531 +0000 UTC m=+61.986255269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls") pod "cluster-samples-operator-65db858894-mqwlr" (UID: "e3c7c7d3-f2da-4987-aadc-5cc832b0c234") : secret "samples-operator-tls" not found Jul 16 12:45:51.991656 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:51.991533 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle podName:15790d2d-298f-4fff-9945-c8bc693a04b1 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:53.991514871 +0000 UTC m=+61.986655623 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle") pod "router-default-75f7d966c4-bb2lf" (UID: "15790d2d-298f-4fff-9945-c8bc693a04b1") : configmap references non-existent config key: service-ca.crt Jul 16 12:45:51.991656 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:51.991602 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 12:45:51.991656 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:51.991634 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs podName:15790d2d-298f-4fff-9945-c8bc693a04b1 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:53.991623235 +0000 UTC m=+61.986763975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs") pod "router-default-75f7d966c4-bb2lf" (UID: "15790d2d-298f-4fff-9945-c8bc693a04b1") : secret "router-metrics-certs-default" not found Jul 16 12:45:52.092107 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:52.092072 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c7ng7\" (UID: \"7a6c1393-8665-4d4a-9238-77f24e498e50\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:45:52.092399 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:52.092379 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 12:45:52.092505 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:52.092450 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls podName:7a6c1393-8665-4d4a-9238-77f24e498e50 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:54.092433309 +0000 UTC m=+62.087574050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-c7ng7" (UID: "7a6c1393-8665-4d4a-9238-77f24e498e50") : secret "cluster-monitoring-operator-tls" not found Jul 16 12:45:52.772281 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:52.772244 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4bdr" Jul 16 12:45:54.011044 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:54.011007 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:54.011044 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:54.011046 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:54.011516 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:54.011086 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mqwlr\" (UID: \"e3c7c7d3-f2da-4987-aadc-5cc832b0c234\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" Jul 16 12:45:54.011516 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:54.011189 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle podName:15790d2d-298f-4fff-9945-c8bc693a04b1 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:58.011171922 +0000 UTC m=+66.006312659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle") pod "router-default-75f7d966c4-bb2lf" (UID: "15790d2d-298f-4fff-9945-c8bc693a04b1") : configmap references non-existent config key: service-ca.crt Jul 16 12:45:54.011516 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:54.011208 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 12:45:54.011516 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:54.011259 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 12:45:54.011516 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:54.011304 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs podName:15790d2d-298f-4fff-9945-c8bc693a04b1 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:58.011285372 +0000 UTC m=+66.006426113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs") pod "router-default-75f7d966c4-bb2lf" (UID: "15790d2d-298f-4fff-9945-c8bc693a04b1") : secret "router-metrics-certs-default" not found Jul 16 12:45:54.011516 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:54.011324 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls podName:e3c7c7d3-f2da-4987-aadc-5cc832b0c234 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:58.011315149 +0000 UTC m=+66.006455892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls") pod "cluster-samples-operator-65db858894-mqwlr" (UID: "e3c7c7d3-f2da-4987-aadc-5cc832b0c234") : secret "samples-operator-tls" not found Jul 16 12:45:54.112584 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:54.112500 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c7ng7\" (UID: \"7a6c1393-8665-4d4a-9238-77f24e498e50\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:45:54.112751 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:54.112661 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 12:45:54.112751 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:54.112732 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls podName:7a6c1393-8665-4d4a-9238-77f24e498e50 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:58.112713481 +0000 UTC m=+66.107854223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-c7ng7" (UID: "7a6c1393-8665-4d4a-9238-77f24e498e50") : secret "cluster-monitoring-operator-tls" not found Jul 16 12:45:54.844145 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:54.844103 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc" event={"ID":"5e27e160-a28d-4523-a137-fb415f779ba5","Type":"ContainerStarted","Data":"ebf1a86ad40cbec3f17acd457ecac2d980ba221d7986b66ba1041e94e6de1326"} Jul 16 12:45:54.845599 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:54.845572 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" event={"ID":"73045a2a-1f69-48c1-96f1-54b941192680","Type":"ContainerStarted","Data":"a815cfee93855fdc6608884a05550067a5612ea44dfd7d417fe5f1059e61aeae"} Jul 16 12:45:54.862713 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:54.862671 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc" podStartSLOduration=1.919073198 podStartE2EDuration="4.86265644s" totalImagesPullingTime="2.943583242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:54.86204859 +0000 UTC m=+62.857189375" watchObservedRunningTime="2026-07-16 12:45:54.86265644 +0000 UTC m=+62.857797200" Jul 16 12:45:54.883773 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:54.883712 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" podStartSLOduration=1.83240614 podStartE2EDuration="4.883685619s" totalImagesPullingTime="3.051279479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:54.882960979 +0000 UTC m=+62.878101738" watchObservedRunningTime="2026-07-16 12:45:54.883685619 +0000 UTC m=+62.878826378" Jul 16 12:45:55.209216 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.207479 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hxrcl"] Jul 16 12:45:55.211418 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.211399 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:55.214083 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.214066 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 12:45:55.214201 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.214137 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g9fmr\"" Jul 16 12:45:55.214308 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.214287 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 12:45:55.227045 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.227022 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hxrcl"] Jul 16 12:45:55.322343 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.322309 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0031d292-78ee-4ea2-b31f-adb988b6d5e3-crio-socket\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:55.322508 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.322357 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:55.322508 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.322464 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:55.322626 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.322522 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bltd\" (UniqueName: \"kubernetes.io/projected/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-api-access-9bltd\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:55.322626 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.322553 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0031d292-78ee-4ea2-b31f-adb988b6d5e3-data-volume\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:55.422955 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.422925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0031d292-78ee-4ea2-b31f-adb988b6d5e3-crio-socket\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:55.423134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.422971 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:55.423134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.423018 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:55.423134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.423044 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bltd\" (UniqueName: \"kubernetes.io/projected/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-api-access-9bltd\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:55.423134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.423070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0031d292-78ee-4ea2-b31f-adb988b6d5e3-data-volume\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:55.423364 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.423067 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0031d292-78ee-4ea2-b31f-adb988b6d5e3-crio-socket\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:55.423364 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:55.423145 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:45:55.423364 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:55.423151 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:45:55.423364 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:55.423217 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm podName:0031d292-78ee-4ea2-b31f-adb988b6d5e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:55.923201831 +0000 UTC m=+63.918342568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxrcl" (UID: "0031d292-78ee-4ea2-b31f-adb988b6d5e3") : configmap "kube-rbac-proxy" not found Jul 16 12:45:55.423364 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:55.423251 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls podName:0031d292-78ee-4ea2-b31f-adb988b6d5e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:55.923224729 +0000 UTC m=+63.918365467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxrcl" (UID: "0031d292-78ee-4ea2-b31f-adb988b6d5e3") : secret "insights-runtime-extractor-tls" not found Jul 16 12:45:55.423566 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.423363 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0031d292-78ee-4ea2-b31f-adb988b6d5e3-data-volume\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:55.437493 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.437470 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bltd\" (UniqueName: \"kubernetes.io/projected/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-api-access-9bltd\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:55.926764 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.926716 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:55.926976 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:55.926826 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:45:55.926976 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:55.926891 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls podName:0031d292-78ee-4ea2-b31f-adb988b6d5e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:56.9268746 +0000 UTC m=+64.922015340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxrcl" (UID: "0031d292-78ee-4ea2-b31f-adb988b6d5e3") : secret "insights-runtime-extractor-tls" not found Jul 16 12:45:55.926976 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:55.926920 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:55.927202 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:55.927015 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:45:55.927202 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:55.927057 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm podName:0031d292-78ee-4ea2-b31f-adb988b6d5e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:56.927046621 +0000 UTC m=+64.922187358 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxrcl" (UID: "0031d292-78ee-4ea2-b31f-adb988b6d5e3") : configmap "kube-rbac-proxy" not found Jul 16 12:45:56.936222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:56.936185 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:56.936615 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:56.936276 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:56.936615 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:56.936335 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:45:56.936615 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:56.936390 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:45:56.936615 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:56.936412 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls podName:0031d292-78ee-4ea2-b31f-adb988b6d5e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:58.936396881 +0000 UTC m=+66.931537619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxrcl" (UID: "0031d292-78ee-4ea2-b31f-adb988b6d5e3") : secret "insights-runtime-extractor-tls" not found Jul 16 12:45:56.936615 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:56.936449 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm podName:0031d292-78ee-4ea2-b31f-adb988b6d5e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:58.936436526 +0000 UTC m=+66.931577280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxrcl" (UID: "0031d292-78ee-4ea2-b31f-adb988b6d5e3") : configmap "kube-rbac-proxy" not found Jul 16 12:45:57.088565 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.088539 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-smgrz_eb3b63d0-7a24-407c-891b-ca52cb88a27c/dns-node-resolver/0.log" Jul 16 12:45:57.239178 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.239092 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs\") pod \"network-metrics-daemon-n2c94\" (UID: \"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe\") " pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:45:57.241710 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.241688 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 12:45:57.249722 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:57.249701 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 16 12:45:57.249822 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:57.249766 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs podName:dd3f2e5f-9499-4947-8d80-160b3ffd0bbe nodeName:}" failed. No retries permitted until 2026-07-16 12:47:01.249745184 +0000 UTC m=+129.244885932 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs") pod "network-metrics-daemon-n2c94" (UID: "dd3f2e5f-9499-4947-8d80-160b3ffd0bbe") : secret "metrics-daemon-secret" not found Jul 16 12:45:57.339709 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.339676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret\") pod \"global-pull-secret-syncer-zdrc5\" (UID: \"5de8db0d-cb13-482f-b0f6-8259d3cb37f9\") " pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:57.342257 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.342220 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 12:45:57.352824 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.352794 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5de8db0d-cb13-482f-b0f6-8259d3cb37f9-original-pull-secret\") pod \"global-pull-secret-syncer-zdrc5\" (UID: \"5de8db0d-cb13-482f-b0f6-8259d3cb37f9\") " pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:57.360784 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.360760 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdrc5" Jul 16 12:45:57.440735 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.440704 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jk4z\" (UniqueName: \"kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z\") pod \"network-check-target-9w6xv\" (UID: \"c4ac2604-e084-4ca8-b7de-bc216ac9d843\") " pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:57.443426 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.443405 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 12:45:57.454164 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.454140 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 12:45:57.465177 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.465153 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jk4z\" (UniqueName: \"kubernetes.io/projected/c4ac2604-e084-4ca8-b7de-bc216ac9d843-kube-api-access-8jk4z\") pod \"network-check-target-9w6xv\" (UID: \"c4ac2604-e084-4ca8-b7de-bc216ac9d843\") " pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:57.490312 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.490219 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zdrc5"] Jul 16 12:45:57.493183 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:45:57.493154 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de8db0d_cb13_482f_b0f6_8259d3cb37f9.slice/crio-77636d5c0022350a3268d5fe2731131caf29beeb28ee38a39462577c67b11b4d WatchSource:0}: Error finding container 77636d5c0022350a3268d5fe2731131caf29beeb28ee38a39462577c67b11b4d: Status 404 returned error can't find the container with id 77636d5c0022350a3268d5fe2731131caf29beeb28ee38a39462577c67b11b4d Jul 16 12:45:57.669254 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.669202 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sbdj2\"" Jul 16 12:45:57.677498 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.677471 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:45:57.805177 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.805147 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9w6xv"] Jul 16 12:45:57.807810 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:45:57.807785 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ac2604_e084_4ca8_b7de_bc216ac9d843.slice/crio-cb5d162acb43a70308907a40b66872928262020ac97f8c29cafffbc0fd1e40f7 WatchSource:0}: Error finding container cb5d162acb43a70308907a40b66872928262020ac97f8c29cafffbc0fd1e40f7: Status 404 returned error can't find the container with id cb5d162acb43a70308907a40b66872928262020ac97f8c29cafffbc0fd1e40f7 Jul 16 12:45:57.853357 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.853318 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9w6xv" event={"ID":"c4ac2604-e084-4ca8-b7de-bc216ac9d843","Type":"ContainerStarted","Data":"cb5d162acb43a70308907a40b66872928262020ac97f8c29cafffbc0fd1e40f7"} Jul 16 12:45:57.854371 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.854351 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zdrc5" event={"ID":"5de8db0d-cb13-482f-b0f6-8259d3cb37f9","Type":"ContainerStarted","Data":"77636d5c0022350a3268d5fe2731131caf29beeb28ee38a39462577c67b11b4d"} Jul 16 12:45:57.888314 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:57.888294 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g64cf_e1549474-0299-489e-af96-767bdf28937a/node-ca/0.log" Jul 16 12:45:58.045016 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:58.044984 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:58.045537 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:58.045023 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:45:58.045537 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:58.045177 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 12:45:58.045537 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:58.045181 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mqwlr\" (UID: \"e3c7c7d3-f2da-4987-aadc-5cc832b0c234\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" Jul 16 12:45:58.045537 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:58.045250 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs podName:15790d2d-298f-4fff-9945-c8bc693a04b1 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:06.045212252 +0000 UTC m=+74.040353080 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs") pod "router-default-75f7d966c4-bb2lf" (UID: "15790d2d-298f-4fff-9945-c8bc693a04b1") : secret "router-metrics-certs-default" not found Jul 16 12:45:58.045537 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:58.045277 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle podName:15790d2d-298f-4fff-9945-c8bc693a04b1 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:06.045266369 +0000 UTC m=+74.040407107 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle") pod "router-default-75f7d966c4-bb2lf" (UID: "15790d2d-298f-4fff-9945-c8bc693a04b1") : configmap references non-existent config key: service-ca.crt Jul 16 12:45:58.045537 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:58.045316 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 12:45:58.045537 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:58.045369 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls podName:e3c7c7d3-f2da-4987-aadc-5cc832b0c234 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:06.045350347 +0000 UTC m=+74.040491086 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls") pod "cluster-samples-operator-65db858894-mqwlr" (UID: "e3c7c7d3-f2da-4987-aadc-5cc832b0c234") : secret "samples-operator-tls" not found Jul 16 12:45:58.146167 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:58.146136 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c7ng7\" (UID: \"7a6c1393-8665-4d4a-9238-77f24e498e50\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:45:58.146366 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:58.146349 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 12:45:58.146439 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:58.146432 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls podName:7a6c1393-8665-4d4a-9238-77f24e498e50 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:06.146410976 +0000 UTC m=+74.141551733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-c7ng7" (UID: "7a6c1393-8665-4d4a-9238-77f24e498e50") : secret "cluster-monitoring-operator-tls" not found Jul 16 12:45:58.489303 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:58.489160 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6n5wk_427c004b-ed80-4024-88ab-b817224073b4/serve-healthcheck-canary/0.log" Jul 16 12:45:58.955242 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:58.954501 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:58.955242 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:58.954580 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:45:58.955242 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:58.954728 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:45:58.955242 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:58.954782 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm podName:0031d292-78ee-4ea2-b31f-adb988b6d5e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:02.95476861 +0000 UTC m=+70.949909348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxrcl" (UID: "0031d292-78ee-4ea2-b31f-adb988b6d5e3") : configmap "kube-rbac-proxy" not found Jul 16 12:45:58.955242 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:58.955152 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:45:58.955242 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:58.955194 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls podName:0031d292-78ee-4ea2-b31f-adb988b6d5e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:02.955178474 +0000 UTC m=+70.950319213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxrcl" (UID: "0031d292-78ee-4ea2-b31f-adb988b6d5e3") : secret "insights-runtime-extractor-tls" not found Jul 16 12:45:59.258889 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:59.258435 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:45:59.258889 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:45:59.258563 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:45:59.258889 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:59.258629 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:45:59.258889 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:59.258655 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55c4f54f86-chtl8: secret "image-registry-tls" not found Jul 16 12:45:59.258889 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:59.258712 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls podName:9eb17f11-3a02-4674-b023-7b781b393e38 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:31.258697229 +0000 UTC m=+99.253837967 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls") pod "image-registry-55c4f54f86-chtl8" (UID: "9eb17f11-3a02-4674-b023-7b781b393e38") : secret "image-registry-tls" not found Jul 16 12:45:59.258889 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:59.258759 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:45:59.258889 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:45:59.258811 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls podName:bd384a9f-61e7-4509-8346-5d345021a603 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:31.258793559 +0000 UTC m=+99.253934313 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls") pod "dns-default-5dxp9" (UID: "bd384a9f-61e7-4509-8346-5d345021a603") : secret "dns-default-metrics-tls" not found Jul 16 12:46:02.869589 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:02.869543 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9w6xv" event={"ID":"c4ac2604-e084-4ca8-b7de-bc216ac9d843","Type":"ContainerStarted","Data":"f34b61b20202e7e8939deac980c1a1f7ab6341f859786a21ea8ba403e7c7bbfd"} Jul 16 12:46:02.870021 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:02.869628 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:46:02.871040 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:02.871019 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zdrc5" event={"ID":"5de8db0d-cb13-482f-b0f6-8259d3cb37f9","Type":"ContainerStarted","Data":"9a17fc5f918cacc18cf4c0a2c615662adef6bfd51ba5ec9a25269995f373bc2e"} Jul 16 12:46:02.890409 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:02.890364 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9w6xv" podStartSLOduration=66.544362356 podStartE2EDuration="1m10.890351997s" totalImagesPullingTime="4.345989641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:02.888520179 +0000 UTC m=+70.883660952" watchObservedRunningTime="2026-07-16 12:46:02.890351997 +0000 UTC m=+70.885492756" Jul 16 12:46:02.995844 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:02.995809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:46:02.996007 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:02.995930 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:46:02.996007 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:02.995978 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls podName:0031d292-78ee-4ea2-b31f-adb988b6d5e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:10.99596411 +0000 UTC m=+78.991104848 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxrcl" (UID: "0031d292-78ee-4ea2-b31f-adb988b6d5e3") : secret "insights-runtime-extractor-tls" not found Jul 16 12:46:02.996007 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:02.995999 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:46:02.996170 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:02.996086 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:46:02.996170 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:02.996114 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm podName:0031d292-78ee-4ea2-b31f-adb988b6d5e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:10.996105301 +0000 UTC m=+78.991246038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxrcl" (UID: "0031d292-78ee-4ea2-b31f-adb988b6d5e3") : configmap "kube-rbac-proxy" not found Jul 16 12:46:06.122146 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:06.122112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:46:06.122146 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:06.122146 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:46:06.122660 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:06.122182 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mqwlr\" (UID: \"e3c7c7d3-f2da-4987-aadc-5cc832b0c234\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" Jul 16 12:46:06.122660 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:06.122296 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle podName:15790d2d-298f-4fff-9945-c8bc693a04b1 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:22.122278933 +0000 UTC m=+90.117419670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle") pod "router-default-75f7d966c4-bb2lf" (UID: "15790d2d-298f-4fff-9945-c8bc693a04b1") : configmap references non-existent config key: service-ca.crt Jul 16 12:46:06.122660 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:06.122297 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 12:46:06.122660 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:06.122333 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs podName:15790d2d-298f-4fff-9945-c8bc693a04b1 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:22.1223256 +0000 UTC m=+90.117466338 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs") pod "router-default-75f7d966c4-bb2lf" (UID: "15790d2d-298f-4fff-9945-c8bc693a04b1") : secret "router-metrics-certs-default" not found Jul 16 12:46:06.124637 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:06.124615 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3c7c7d3-f2da-4987-aadc-5cc832b0c234-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mqwlr\" (UID: \"e3c7c7d3-f2da-4987-aadc-5cc832b0c234\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" Jul 16 12:46:06.223473 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:06.223438 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c7ng7\" (UID: \"7a6c1393-8665-4d4a-9238-77f24e498e50\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:46:06.223633 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:06.223582 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 12:46:06.223672 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:06.223659 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls podName:7a6c1393-8665-4d4a-9238-77f24e498e50 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:22.223645126 +0000 UTC m=+90.218785863 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-c7ng7" (UID: "7a6c1393-8665-4d4a-9238-77f24e498e50") : secret "cluster-monitoring-operator-tls" not found Jul 16 12:46:06.274805 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:06.274778 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h4nj6\"" Jul 16 12:46:06.283260 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:06.283218 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" Jul 16 12:46:06.410144 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:06.410054 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zdrc5" podStartSLOduration=68.74575485 podStartE2EDuration="1m13.410008009s" totalImagesPullingTime="4.664253159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:02.909522233 +0000 UTC m=+70.904663012" watchObservedRunningTime="2026-07-16 12:46:06.410008009 +0000 UTC m=+74.405148768" Jul 16 12:46:06.410359 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:06.410343 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr"] Jul 16 12:46:06.883952 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:06.883919 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" event={"ID":"e3c7c7d3-f2da-4987-aadc-5cc832b0c234","Type":"ContainerStarted","Data":"5c0366e762ca9570881b272059ce579f8fbdb62a3025fa3aee678f3ed1dde141"} Jul 16 12:46:08.892835 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:08.892798 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" event={"ID":"e3c7c7d3-f2da-4987-aadc-5cc832b0c234","Type":"ContainerStarted","Data":"d2c390704a67fcc7a3d4e8e5a69db569d7dbdd368d08b15fe9c6e72174fa7dd3"} Jul 16 12:46:08.893252 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:08.892851 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" event={"ID":"e3c7c7d3-f2da-4987-aadc-5cc832b0c234","Type":"ContainerStarted","Data":"a34e81426f7cb508ed85b191c8ce4faa06de60a29528b1e1546087f90411560b"} Jul 16 12:46:08.927824 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:08.927778 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mqwlr" podStartSLOduration=17.384031746 podStartE2EDuration="18.927763411s" totalImagesPullingTime="1.543731665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:08.926861502 +0000 UTC m=+76.922002263" watchObservedRunningTime="2026-07-16 12:46:08.927763411 +0000 UTC m=+76.922904170" Jul 16 12:46:11.068971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:11.068918 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:46:11.069505 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:11.069043 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:46:11.069505 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:11.069143 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:46:11.069505 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:11.069194 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls podName:0031d292-78ee-4ea2-b31f-adb988b6d5e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:27.06918135 +0000 UTC m=+95.064322087 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxrcl" (UID: "0031d292-78ee-4ea2-b31f-adb988b6d5e3") : secret "insights-runtime-extractor-tls" not found Jul 16 12:46:11.069703 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:11.069602 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0031d292-78ee-4ea2-b31f-adb988b6d5e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:46:17.728245 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:17.728113 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-gz4px"] Jul 16 12:46:17.732532 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:17.732515 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-gz4px" Jul 16 12:46:17.735081 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:17.735056 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lkdzs\"" Jul 16 12:46:17.735081 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:17.735058 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 12:46:17.735377 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:17.735363 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 12:46:17.747474 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:17.747450 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-gz4px"] Jul 16 12:46:17.818782 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:17.818755 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2g4l\" (UniqueName: \"kubernetes.io/projected/180b114f-387d-46cf-a493-7ad185105df3-kube-api-access-f2g4l\") pod \"downloads-68c9cbb497-gz4px\" (UID: \"180b114f-387d-46cf-a493-7ad185105df3\") " pod="openshift-console/downloads-68c9cbb497-gz4px" Jul 16 12:46:17.922281 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:17.922252 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2g4l\" (UniqueName: \"kubernetes.io/projected/180b114f-387d-46cf-a493-7ad185105df3-kube-api-access-f2g4l\") pod \"downloads-68c9cbb497-gz4px\" (UID: \"180b114f-387d-46cf-a493-7ad185105df3\") " pod="openshift-console/downloads-68c9cbb497-gz4px" Jul 16 12:46:17.937311 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:17.937284 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2g4l\" (UniqueName: \"kubernetes.io/projected/180b114f-387d-46cf-a493-7ad185105df3-kube-api-access-f2g4l\") pod \"downloads-68c9cbb497-gz4px\" (UID: \"180b114f-387d-46cf-a493-7ad185105df3\") " pod="openshift-console/downloads-68c9cbb497-gz4px" Jul 16 12:46:18.041139 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:18.041113 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-gz4px" Jul 16 12:46:18.180384 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:18.180294 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-gz4px"] Jul 16 12:46:18.183808 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:46:18.183776 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180b114f_387d_46cf_a493_7ad185105df3.slice/crio-0e9a19db4af27ce1af7f9899f6483830d1de2b9da86f6b1469e3ff11ee83fdf9 WatchSource:0}: Error finding container 0e9a19db4af27ce1af7f9899f6483830d1de2b9da86f6b1469e3ff11ee83fdf9: Status 404 returned error can't find the container with id 0e9a19db4af27ce1af7f9899f6483830d1de2b9da86f6b1469e3ff11ee83fdf9 Jul 16 12:46:18.927283 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:18.927241 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-gz4px" event={"ID":"180b114f-387d-46cf-a493-7ad185105df3","Type":"ContainerStarted","Data":"0e9a19db4af27ce1af7f9899f6483830d1de2b9da86f6b1469e3ff11ee83fdf9"} Jul 16 12:46:22.169336 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:22.169301 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:46:22.169336 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:22.169337 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:46:22.170105 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:22.170079 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15790d2d-298f-4fff-9945-c8bc693a04b1-service-ca-bundle\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:46:22.171937 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:22.171907 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15790d2d-298f-4fff-9945-c8bc693a04b1-metrics-certs\") pod \"router-default-75f7d966c4-bb2lf\" (UID: \"15790d2d-298f-4fff-9945-c8bc693a04b1\") " pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:46:22.270713 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:22.270672 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c7ng7\" (UID: \"7a6c1393-8665-4d4a-9238-77f24e498e50\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:46:22.273508 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:22.273478 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a6c1393-8665-4d4a-9238-77f24e498e50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c7ng7\" (UID: \"7a6c1393-8665-4d4a-9238-77f24e498e50\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:46:22.388829 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:22.388797 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jmjpw\"" Jul 16 12:46:22.396879 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:22.396858 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:46:22.490471 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:22.490447 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6kbfr\"" Jul 16 12:46:22.498290 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:22.498265 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" Jul 16 12:46:22.556438 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:22.555915 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75f7d966c4-bb2lf"] Jul 16 12:46:22.560886 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:46:22.560755 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15790d2d_298f_4fff_9945_c8bc693a04b1.slice/crio-9b9f5ac79311961e1013045e9f01e795ea16b92467bc4865662bea9ae666b11f WatchSource:0}: Error finding container 9b9f5ac79311961e1013045e9f01e795ea16b92467bc4865662bea9ae666b11f: Status 404 returned error can't find the container with id 9b9f5ac79311961e1013045e9f01e795ea16b92467bc4865662bea9ae666b11f Jul 16 12:46:22.653849 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:22.653812 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7"] Jul 16 12:46:22.657129 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:46:22.657100 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6c1393_8665_4d4a_9238_77f24e498e50.slice/crio-655eb69c8756a03d57725c05f7ec249df3f4067e94c17ef7efc4da3d2d4aa1b4 WatchSource:0}: Error finding container 655eb69c8756a03d57725c05f7ec249df3f4067e94c17ef7efc4da3d2d4aa1b4: Status 404 returned error can't find the container with id 655eb69c8756a03d57725c05f7ec249df3f4067e94c17ef7efc4da3d2d4aa1b4 Jul 16 12:46:22.942424 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:22.942388 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" event={"ID":"7a6c1393-8665-4d4a-9238-77f24e498e50","Type":"ContainerStarted","Data":"655eb69c8756a03d57725c05f7ec249df3f4067e94c17ef7efc4da3d2d4aa1b4"} Jul 16 12:46:22.944040 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:22.944000 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75f7d966c4-bb2lf" event={"ID":"15790d2d-298f-4fff-9945-c8bc693a04b1","Type":"ContainerStarted","Data":"c8625a23c3f74d131756b6b73189cedfd9bae97a11c9d5a8f29d2f1cf560c761"} Jul 16 12:46:22.944169 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:22.944077 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75f7d966c4-bb2lf" event={"ID":"15790d2d-298f-4fff-9945-c8bc693a04b1","Type":"ContainerStarted","Data":"9b9f5ac79311961e1013045e9f01e795ea16b92467bc4865662bea9ae666b11f"} Jul 16 12:46:22.976132 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:22.976045 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-75f7d966c4-bb2lf" podStartSLOduration=32.976026471 podStartE2EDuration="32.976026471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:22.97496909 +0000 UTC m=+90.970109841" watchObservedRunningTime="2026-07-16 12:46:22.976026471 +0000 UTC m=+90.971167223" Jul 16 12:46:23.397290 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:23.397252 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:46:23.400103 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:23.400079 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:46:23.948591 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:23.948560 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:46:23.950048 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:23.949994 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-75f7d966c4-bb2lf" Jul 16 12:46:24.954369 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:24.954320 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" event={"ID":"7a6c1393-8665-4d4a-9238-77f24e498e50","Type":"ContainerStarted","Data":"b0686786c74a3b8e60f91dcde2ee314f29c79f02445ff49f6862baf9e41227a4"} Jul 16 12:46:25.431916 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:25.431865 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c7ng7" podStartSLOduration=33.252177765 podStartE2EDuration="35.431843131s" totalImagesPullingTime="2.179665366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:24.986480276 +0000 UTC m=+92.981621036" watchObservedRunningTime="2026-07-16 12:46:25.431843131 +0000 UTC m=+93.426983924" Jul 16 12:46:25.432103 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:25.432016 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8lnf7"] Jul 16 12:46:25.434346 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:25.434321 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8lnf7" Jul 16 12:46:25.436503 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:25.436480 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 16 12:46:25.436646 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:25.436551 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-d5lns\"" Jul 16 12:46:25.449856 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:25.449831 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8lnf7"] Jul 16 12:46:25.496850 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:25.496820 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d2fafe97-2a61-4045-925c-89129f1c1f72-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-8lnf7\" (UID: \"d2fafe97-2a61-4045-925c-89129f1c1f72\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8lnf7" Jul 16 12:46:25.597931 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:25.597897 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d2fafe97-2a61-4045-925c-89129f1c1f72-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-8lnf7\" (UID: \"d2fafe97-2a61-4045-925c-89129f1c1f72\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8lnf7" Jul 16 12:46:25.598102 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:25.598056 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 16 12:46:25.598164 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:25.598136 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2fafe97-2a61-4045-925c-89129f1c1f72-tls-certificates podName:d2fafe97-2a61-4045-925c-89129f1c1f72 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:26.098113833 +0000 UTC m=+94.093254579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/d2fafe97-2a61-4045-925c-89129f1c1f72-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-8lnf7" (UID: "d2fafe97-2a61-4045-925c-89129f1c1f72") : secret "prometheus-operator-admission-webhook-tls" not found Jul 16 12:46:26.102819 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.102786 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d2fafe97-2a61-4045-925c-89129f1c1f72-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-8lnf7\" (UID: \"d2fafe97-2a61-4045-925c-89129f1c1f72\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8lnf7" Jul 16 12:46:26.106166 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.106137 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d2fafe97-2a61-4045-925c-89129f1c1f72-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-8lnf7\" (UID: \"d2fafe97-2a61-4045-925c-89129f1c1f72\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8lnf7" Jul 16 12:46:26.344644 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.344612 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8lnf7" Jul 16 12:46:26.493525 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.493494 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8lnf7"] Jul 16 12:46:26.496797 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:46:26.496768 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2fafe97_2a61_4045_925c_89129f1c1f72.slice/crio-ab96e632f0a8bed426f9f421bd3e6027f82c9468f15ed96d8953730fd3018b3d WatchSource:0}: Error finding container ab96e632f0a8bed426f9f421bd3e6027f82c9468f15ed96d8953730fd3018b3d: Status 404 returned error can't find the container with id ab96e632f0a8bed426f9f421bd3e6027f82c9468f15ed96d8953730fd3018b3d Jul 16 12:46:26.553735 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.553698 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57457978f4-hgqb2"] Jul 16 12:46:26.556054 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.556026 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.558605 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.558573 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 12:46:26.558717 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.558641 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 12:46:26.558717 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.558667 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 12:46:26.559737 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.559520 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-678j9\"" Jul 16 12:46:26.559737 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.559692 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 12:46:26.559902 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.559790 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 12:46:26.569503 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.569477 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57457978f4-hgqb2"] Jul 16 12:46:26.607896 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.607770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe654dab-4eec-4c27-be43-5fe243225ef9-console-serving-cert\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.608033 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.607981 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-oauth-serving-cert\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.608096 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.608050 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvp8g\" (UniqueName: \"kubernetes.io/projected/fe654dab-4eec-4c27-be43-5fe243225ef9-kube-api-access-gvp8g\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.608096 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.608088 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe654dab-4eec-4c27-be43-5fe243225ef9-console-oauth-config\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.608195 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.608125 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-console-config\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.608195 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.608166 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-service-ca\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.708822 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.708784 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-oauth-serving-cert\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.708985 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.708854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvp8g\" (UniqueName: \"kubernetes.io/projected/fe654dab-4eec-4c27-be43-5fe243225ef9-kube-api-access-gvp8g\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.708985 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.708883 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe654dab-4eec-4c27-be43-5fe243225ef9-console-oauth-config\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.708985 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.708910 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-console-config\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.708985 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.708940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-service-ca\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.709216 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.708999 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe654dab-4eec-4c27-be43-5fe243225ef9-console-serving-cert\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.709656 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.709624 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-oauth-serving-cert\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.709784 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.709738 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-service-ca\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.709870 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.709847 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-console-config\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.712180 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.712157 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe654dab-4eec-4c27-be43-5fe243225ef9-console-oauth-config\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.712470 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.712445 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe654dab-4eec-4c27-be43-5fe243225ef9-console-serving-cert\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.719341 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.719316 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvp8g\" (UniqueName: \"kubernetes.io/projected/fe654dab-4eec-4c27-be43-5fe243225ef9-kube-api-access-gvp8g\") pod \"console-57457978f4-hgqb2\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.868643 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.868563 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:26.965395 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:26.965279 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8lnf7" event={"ID":"d2fafe97-2a61-4045-925c-89129f1c1f72","Type":"ContainerStarted","Data":"ab96e632f0a8bed426f9f421bd3e6027f82c9468f15ed96d8953730fd3018b3d"} Jul 16 12:46:27.013706 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:27.013656 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57457978f4-hgqb2"] Jul 16 12:46:27.017434 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:46:27.017402 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe654dab_4eec_4c27_be43_5fe243225ef9.slice/crio-c2985160fe7b5ca756ee50fc15a8820b654aa03a40f468ea4ded1a46bd7e868e WatchSource:0}: Error finding container c2985160fe7b5ca756ee50fc15a8820b654aa03a40f468ea4ded1a46bd7e868e: Status 404 returned error can't find the container with id c2985160fe7b5ca756ee50fc15a8820b654aa03a40f468ea4ded1a46bd7e868e Jul 16 12:46:27.112561 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:27.112521 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:46:27.115634 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:27.115604 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0031d292-78ee-4ea2-b31f-adb988b6d5e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxrcl\" (UID: \"0031d292-78ee-4ea2-b31f-adb988b6d5e3\") " pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:46:27.320224 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:27.320191 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hxrcl" Jul 16 12:46:27.970378 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:27.970340 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57457978f4-hgqb2" event={"ID":"fe654dab-4eec-4c27-be43-5fe243225ef9","Type":"ContainerStarted","Data":"c2985160fe7b5ca756ee50fc15a8820b654aa03a40f468ea4ded1a46bd7e868e"} Jul 16 12:46:31.354146 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:31.354106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:46:31.354636 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:31.354188 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:46:31.357152 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:31.357129 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd384a9f-61e7-4509-8346-5d345021a603-metrics-tls\") pod \"dns-default-5dxp9\" (UID: \"bd384a9f-61e7-4509-8346-5d345021a603\") " pod="openshift-dns/dns-default-5dxp9" Jul 16 12:46:31.357305 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:31.357218 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls\") pod \"image-registry-55c4f54f86-chtl8\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:46:31.544347 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:31.544312 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rr49k\"" Jul 16 12:46:31.551590 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:31.551561 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lnsbk\"" Jul 16 12:46:31.551746 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:31.551658 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:46:31.559581 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:31.559551 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5dxp9" Jul 16 12:46:33.877073 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:33.877035 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9w6xv" Jul 16 12:46:35.124096 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:35.124071 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hxrcl"] Jul 16 12:46:35.128532 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:46:35.128495 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0031d292_78ee_4ea2_b31f_adb988b6d5e3.slice/crio-be3fdf9fa2dc8b3841e257c1a1e13296d5f8560fca1984e201823579e5224263 WatchSource:0}: Error finding container be3fdf9fa2dc8b3841e257c1a1e13296d5f8560fca1984e201823579e5224263: Status 404 returned error can't find the container with id be3fdf9fa2dc8b3841e257c1a1e13296d5f8560fca1984e201823579e5224263 Jul 16 12:46:35.144140 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:35.144112 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5dxp9"] Jul 16 12:46:35.147450 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:46:35.147420 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd384a9f_61e7_4509_8346_5d345021a603.slice/crio-39a1c2d07945c56ffbb4a29d2a9a5fd18a5bb98b30ff4c0125149bfbd725bedb WatchSource:0}: Error finding container 39a1c2d07945c56ffbb4a29d2a9a5fd18a5bb98b30ff4c0125149bfbd725bedb: Status 404 returned error can't find the container with id 39a1c2d07945c56ffbb4a29d2a9a5fd18a5bb98b30ff4c0125149bfbd725bedb Jul 16 12:46:35.166107 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:35.166009 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55c4f54f86-chtl8"] Jul 16 12:46:35.168872 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:46:35.168842 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb17f11_3a02_4674_b023_7b781b393e38.slice/crio-07b15dd64d5b5dffb4113d4e32fe86317fb99632be546990158eb14f5d711e09 WatchSource:0}: Error finding container 07b15dd64d5b5dffb4113d4e32fe86317fb99632be546990158eb14f5d711e09: Status 404 returned error can't find the container with id 07b15dd64d5b5dffb4113d4e32fe86317fb99632be546990158eb14f5d711e09 Jul 16 12:46:36.009021 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.008138 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" event={"ID":"9eb17f11-3a02-4674-b023-7b781b393e38","Type":"ContainerStarted","Data":"50462a6d2a2961156d457de71e6a6c7eb00402b9329bf5fd7d691a18be77ce56"} Jul 16 12:46:36.009021 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.008202 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" event={"ID":"9eb17f11-3a02-4674-b023-7b781b393e38","Type":"ContainerStarted","Data":"07b15dd64d5b5dffb4113d4e32fe86317fb99632be546990158eb14f5d711e09"} Jul 16 12:46:36.009021 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.008955 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:46:36.010690 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.010587 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxrcl" event={"ID":"0031d292-78ee-4ea2-b31f-adb988b6d5e3","Type":"ContainerStarted","Data":"be3fdf9fa2dc8b3841e257c1a1e13296d5f8560fca1984e201823579e5224263"} Jul 16 12:46:36.012814 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.012785 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-gz4px" event={"ID":"180b114f-387d-46cf-a493-7ad185105df3","Type":"ContainerStarted","Data":"4e7bb685bee7a8e1a348bec1966377d2b0aabc1987c53140bb841275052b5955"} Jul 16 12:46:36.013511 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.013489 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-gz4px" Jul 16 12:46:36.015857 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.015830 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5dxp9" event={"ID":"bd384a9f-61e7-4509-8346-5d345021a603","Type":"ContainerStarted","Data":"39a1c2d07945c56ffbb4a29d2a9a5fd18a5bb98b30ff4c0125149bfbd725bedb"} Jul 16 12:46:36.017681 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.017655 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8lnf7" event={"ID":"d2fafe97-2a61-4045-925c-89129f1c1f72","Type":"ContainerStarted","Data":"7258f3a534625c84ad71cf763379410ec2dfd420deba5f73f742eb31901f18f9"} Jul 16 12:46:36.018382 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.018360 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8lnf7" Jul 16 12:46:36.024569 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.024522 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8lnf7" Jul 16 12:46:36.025217 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.025193 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-gz4px" Jul 16 12:46:36.034726 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.034679 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" podStartSLOduration=103.034664782 podStartE2EDuration="1m43.034664782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:36.032595196 +0000 UTC m=+104.027735972" watchObservedRunningTime="2026-07-16 12:46:36.034664782 +0000 UTC m=+104.029805544" Jul 16 12:46:36.051998 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.051945 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8lnf7" podStartSLOduration=2.626697287 podStartE2EDuration="11.051925329s" totalImagesPullingTime="8.425228042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:36.050887431 +0000 UTC m=+104.046028196" watchObservedRunningTime="2026-07-16 12:46:36.051925329 +0000 UTC m=+104.047066090" Jul 16 12:46:36.072611 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.072515 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-gz4px" podStartSLOduration=2.31993117 podStartE2EDuration="19.072493209s" totalImagesPullingTime="16.752562039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:36.071614329 +0000 UTC m=+104.066755091" watchObservedRunningTime="2026-07-16 12:46:36.072493209 +0000 UTC m=+104.067633968" Jul 16 12:46:36.555496 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.555459 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-hmrt5"] Jul 16 12:46:36.586179 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.586140 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-hmrt5"] Jul 16 12:46:36.586431 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.586315 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" Jul 16 12:46:36.590044 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.589952 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 16 12:46:36.590307 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.590288 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-98f5p\"" Jul 16 12:46:36.591541 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.590908 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 16 12:46:36.591541 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.591142 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 12:46:36.698791 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.698471 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/acbfde83-0959-4f8e-a53e-33c800463dec-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-hmrt5\" (UID: \"acbfde83-0959-4f8e-a53e-33c800463dec\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" Jul 16 12:46:36.698791 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.698540 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acbfde83-0959-4f8e-a53e-33c800463dec-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-hmrt5\" (UID: \"acbfde83-0959-4f8e-a53e-33c800463dec\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" Jul 16 12:46:36.698791 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.698567 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htl9h\" (UniqueName: \"kubernetes.io/projected/acbfde83-0959-4f8e-a53e-33c800463dec-kube-api-access-htl9h\") pod \"prometheus-operator-64785b6675-hmrt5\" (UID: \"acbfde83-0959-4f8e-a53e-33c800463dec\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" Jul 16 12:46:36.698791 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.698604 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acbfde83-0959-4f8e-a53e-33c800463dec-metrics-client-ca\") pod \"prometheus-operator-64785b6675-hmrt5\" (UID: \"acbfde83-0959-4f8e-a53e-33c800463dec\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" Jul 16 12:46:36.801667 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.800047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/acbfde83-0959-4f8e-a53e-33c800463dec-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-hmrt5\" (UID: \"acbfde83-0959-4f8e-a53e-33c800463dec\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" Jul 16 12:46:36.801667 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.800127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acbfde83-0959-4f8e-a53e-33c800463dec-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-hmrt5\" (UID: \"acbfde83-0959-4f8e-a53e-33c800463dec\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" Jul 16 12:46:36.801667 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.800160 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htl9h\" (UniqueName: \"kubernetes.io/projected/acbfde83-0959-4f8e-a53e-33c800463dec-kube-api-access-htl9h\") pod \"prometheus-operator-64785b6675-hmrt5\" (UID: \"acbfde83-0959-4f8e-a53e-33c800463dec\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" Jul 16 12:46:36.801667 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.800193 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acbfde83-0959-4f8e-a53e-33c800463dec-metrics-client-ca\") pod \"prometheus-operator-64785b6675-hmrt5\" (UID: \"acbfde83-0959-4f8e-a53e-33c800463dec\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" Jul 16 12:46:36.801667 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.800906 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acbfde83-0959-4f8e-a53e-33c800463dec-metrics-client-ca\") pod \"prometheus-operator-64785b6675-hmrt5\" (UID: \"acbfde83-0959-4f8e-a53e-33c800463dec\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" Jul 16 12:46:36.801667 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:36.801017 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 16 12:46:36.801667 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:36.801079 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acbfde83-0959-4f8e-a53e-33c800463dec-prometheus-operator-tls podName:acbfde83-0959-4f8e-a53e-33c800463dec nodeName:}" failed. No retries permitted until 2026-07-16 12:46:37.301061004 +0000 UTC m=+105.296201744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/acbfde83-0959-4f8e-a53e-33c800463dec-prometheus-operator-tls") pod "prometheus-operator-64785b6675-hmrt5" (UID: "acbfde83-0959-4f8e-a53e-33c800463dec") : secret "prometheus-operator-tls" not found Jul 16 12:46:36.806991 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.806898 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acbfde83-0959-4f8e-a53e-33c800463dec-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-hmrt5\" (UID: \"acbfde83-0959-4f8e-a53e-33c800463dec\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" Jul 16 12:46:36.816515 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:36.816484 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htl9h\" (UniqueName: \"kubernetes.io/projected/acbfde83-0959-4f8e-a53e-33c800463dec-kube-api-access-htl9h\") pod \"prometheus-operator-64785b6675-hmrt5\" (UID: \"acbfde83-0959-4f8e-a53e-33c800463dec\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" Jul 16 12:46:37.306287 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:37.306241 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/acbfde83-0959-4f8e-a53e-33c800463dec-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-hmrt5\" (UID: \"acbfde83-0959-4f8e-a53e-33c800463dec\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" Jul 16 12:46:37.309278 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:37.309187 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/acbfde83-0959-4f8e-a53e-33c800463dec-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-hmrt5\" (UID: \"acbfde83-0959-4f8e-a53e-33c800463dec\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" Jul 16 12:46:37.502731 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:37.502696 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" Jul 16 12:46:37.694841 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:37.694806 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-hmrt5"] Jul 16 12:46:37.743455 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:46:37.743416 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbfde83_0959_4f8e_a53e_33c800463dec.slice/crio-4801f0c69e479c8dacd86645a63dad70ce1da2eac606bb2e5a005c1cb576e5b8 WatchSource:0}: Error finding container 4801f0c69e479c8dacd86645a63dad70ce1da2eac606bb2e5a005c1cb576e5b8: Status 404 returned error can't find the container with id 4801f0c69e479c8dacd86645a63dad70ce1da2eac606bb2e5a005c1cb576e5b8 Jul 16 12:46:38.033074 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:38.033018 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57457978f4-hgqb2" event={"ID":"fe654dab-4eec-4c27-be43-5fe243225ef9","Type":"ContainerStarted","Data":"17e1b62fa1c646488eecf1b0780f9fca5baf75efb8cc2ce74ee4b508e5a45432"} Jul 16 12:46:38.037932 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:38.037903 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" event={"ID":"acbfde83-0959-4f8e-a53e-33c800463dec","Type":"ContainerStarted","Data":"4801f0c69e479c8dacd86645a63dad70ce1da2eac606bb2e5a005c1cb576e5b8"} Jul 16 12:46:38.061654 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:38.061591 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57457978f4-hgqb2" podStartSLOduration=1.304179532 podStartE2EDuration="12.061567198s" totalImagesPullingTime="10.757387666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:38.05934868 +0000 UTC m=+106.054489447" watchObservedRunningTime="2026-07-16 12:46:38.061567198 +0000 UTC m=+106.056707965" Jul 16 12:46:40.381971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:40.381278 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55c4f54f86-chtl8"] Jul 16 12:46:46.079086 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:46.079050 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxrcl" event={"ID":"0031d292-78ee-4ea2-b31f-adb988b6d5e3","Type":"ContainerStarted","Data":"3dd9e2f9f2f54bec2150e2609dbd0ad914ffa5185fd096103f0348c2a40d3879"} Jul 16 12:46:46.080956 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:46.080931 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" event={"ID":"acbfde83-0959-4f8e-a53e-33c800463dec","Type":"ContainerStarted","Data":"31754759ae3f063842f70a812cdf9f57c3ff42da3b970a1599471da6779b6980"} Jul 16 12:46:46.081074 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:46.080988 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" event={"ID":"acbfde83-0959-4f8e-a53e-33c800463dec","Type":"ContainerStarted","Data":"12a3d634bd12f71a4146d495d6b31fc8367573b01d2975461ceae871e3dfc88e"} Jul 16 12:46:46.082604 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:46.082582 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5dxp9" event={"ID":"bd384a9f-61e7-4509-8346-5d345021a603","Type":"ContainerStarted","Data":"e5b33b66f6a0bf544f3bd47b3fa9db612a6043ec2d41451b539986d865a8a07f"} Jul 16 12:46:46.082695 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:46.082620 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5dxp9" event={"ID":"bd384a9f-61e7-4509-8346-5d345021a603","Type":"ContainerStarted","Data":"b6d5d06b6a8077cdff0e258efa5a81ead5c1eb44e3ffd44901956dcef24aae8e"} Jul 16 12:46:46.082695 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:46.082661 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5dxp9" Jul 16 12:46:46.103040 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:46.102992 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-hmrt5" podStartSLOduration=2.5866787799999997 podStartE2EDuration="10.1029793s" totalImagesPullingTime="7.51630052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:46.101387066 +0000 UTC m=+114.096527823" watchObservedRunningTime="2026-07-16 12:46:46.1029793 +0000 UTC m=+114.098120059" Jul 16 12:46:46.125625 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:46.125587 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5dxp9" podStartSLOduration=68.990144259 podStartE2EDuration="1m19.125573031s" totalImagesPullingTime="10.135428772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:46.124396612 +0000 UTC m=+114.119537369" watchObservedRunningTime="2026-07-16 12:46:46.125573031 +0000 UTC m=+114.120713790" Jul 16 12:46:46.869513 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:46.869469 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:46.869712 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:46.869551 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:46.875395 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:46.875370 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:47.091631 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:47.091603 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:46:48.129094 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.129062 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb"] Jul 16 12:46:48.210555 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.210519 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb"] Jul 16 12:46:48.210738 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.210572 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c6z6g"] Jul 16 12:46:48.210738 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.210667 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" Jul 16 12:46:48.213723 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.213662 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 16 12:46:48.213943 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.213923 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 16 12:46:48.214036 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.213970 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-n8s59\"" Jul 16 12:46:48.250721 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.250688 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.253449 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.253428 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 12:46:48.253585 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.253485 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2nc5p\"" Jul 16 12:46:48.253585 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.253494 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 12:46:48.253585 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.253555 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 12:46:48.314054 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.314020 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5280694-1df9-4c77-ac14-eee54b7cc27f-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-p6swb\" (UID: \"c5280694-1df9-4c77-ac14-eee54b7cc27f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" Jul 16 12:46:48.314223 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.314085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5280694-1df9-4c77-ac14-eee54b7cc27f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-p6swb\" (UID: \"c5280694-1df9-4c77-ac14-eee54b7cc27f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" Jul 16 12:46:48.314223 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.314193 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5280694-1df9-4c77-ac14-eee54b7cc27f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-p6swb\" (UID: \"c5280694-1df9-4c77-ac14-eee54b7cc27f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" Jul 16 12:46:48.314363 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.314224 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfmrm\" (UniqueName: \"kubernetes.io/projected/c5280694-1df9-4c77-ac14-eee54b7cc27f-kube-api-access-tfmrm\") pod \"openshift-state-metrics-84bd68b896-p6swb\" (UID: \"c5280694-1df9-4c77-ac14-eee54b7cc27f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" Jul 16 12:46:48.414900 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.414823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-textfile\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.414900 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.414862 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5280694-1df9-4c77-ac14-eee54b7cc27f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-p6swb\" (UID: \"c5280694-1df9-4c77-ac14-eee54b7cc27f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" Jul 16 12:46:48.414900 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.414886 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfmrm\" (UniqueName: \"kubernetes.io/projected/c5280694-1df9-4c77-ac14-eee54b7cc27f-kube-api-access-tfmrm\") pod \"openshift-state-metrics-84bd68b896-p6swb\" (UID: \"c5280694-1df9-4c77-ac14-eee54b7cc27f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" Jul 16 12:46:48.415172 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.414919 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-wtmp\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.415172 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.414935 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.415172 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:48.415005 2565 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 16 12:46:48.415172 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.415032 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-tls\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.415172 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:48.415088 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5280694-1df9-4c77-ac14-eee54b7cc27f-openshift-state-metrics-tls podName:c5280694-1df9-4c77-ac14-eee54b7cc27f nodeName:}" failed. No retries permitted until 2026-07-16 12:46:48.915066451 +0000 UTC m=+116.910207192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c5280694-1df9-4c77-ac14-eee54b7cc27f-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-p6swb" (UID: "c5280694-1df9-4c77-ac14-eee54b7cc27f") : secret "openshift-state-metrics-tls" not found Jul 16 12:46:48.415462 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.415171 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5280694-1df9-4c77-ac14-eee54b7cc27f-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-p6swb\" (UID: \"c5280694-1df9-4c77-ac14-eee54b7cc27f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" Jul 16 12:46:48.415462 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.415198 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-sys\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.415462 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.415224 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxkd8\" (UniqueName: \"kubernetes.io/projected/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-kube-api-access-mxkd8\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.415462 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.415284 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5280694-1df9-4c77-ac14-eee54b7cc27f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-p6swb\" (UID: \"c5280694-1df9-4c77-ac14-eee54b7cc27f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" Jul 16 12:46:48.415462 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.415320 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-metrics-client-ca\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.415462 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.415368 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.415462 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.415401 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-root\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.415872 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.415852 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5280694-1df9-4c77-ac14-eee54b7cc27f-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-p6swb\" (UID: \"c5280694-1df9-4c77-ac14-eee54b7cc27f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" Jul 16 12:46:48.417984 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.417960 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5280694-1df9-4c77-ac14-eee54b7cc27f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-p6swb\" (UID: \"c5280694-1df9-4c77-ac14-eee54b7cc27f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" Jul 16 12:46:48.442939 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.442901 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfmrm\" (UniqueName: \"kubernetes.io/projected/c5280694-1df9-4c77-ac14-eee54b7cc27f-kube-api-access-tfmrm\") pod \"openshift-state-metrics-84bd68b896-p6swb\" (UID: \"c5280694-1df9-4c77-ac14-eee54b7cc27f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" Jul 16 12:46:48.516873 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.516837 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-sys\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.516873 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.516881 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxkd8\" (UniqueName: \"kubernetes.io/projected/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-kube-api-access-mxkd8\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.517107 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.516912 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-metrics-client-ca\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.517107 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.516948 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.517107 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.516979 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-root\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.517107 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.516987 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-sys\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.517107 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.517011 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-textfile\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.517107 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.517063 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-wtmp\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.517107 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.517084 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.517950 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.517136 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-tls\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.517950 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:48.517270 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 16 12:46:48.517950 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:48.517331 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-tls podName:3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b nodeName:}" failed. No retries permitted until 2026-07-16 12:46:49.017310867 +0000 UTC m=+117.012451626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-tls") pod "node-exporter-c6z6g" (UID: "3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b") : secret "node-exporter-tls" not found Jul 16 12:46:48.517950 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.517429 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-wtmp\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.517950 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.517497 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-root\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.517950 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.517661 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-textfile\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.517950 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.517673 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-metrics-client-ca\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.517950 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.517802 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.519667 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.519647 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.532866 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.532784 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57457978f4-hgqb2"] Jul 16 12:46:48.535286 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.535221 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxkd8\" (UniqueName: \"kubernetes.io/projected/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-kube-api-access-mxkd8\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:48.921571 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.921535 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5280694-1df9-4c77-ac14-eee54b7cc27f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-p6swb\" (UID: \"c5280694-1df9-4c77-ac14-eee54b7cc27f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" Jul 16 12:46:48.923994 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:48.923970 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5280694-1df9-4c77-ac14-eee54b7cc27f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-p6swb\" (UID: \"c5280694-1df9-4c77-ac14-eee54b7cc27f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" Jul 16 12:46:49.022534 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:49.022499 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-tls\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:49.024786 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:49.024759 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b-node-exporter-tls\") pod \"node-exporter-c6z6g\" (UID: \"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b\") " pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:49.122288 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:49.122262 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" Jul 16 12:46:49.161696 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:49.161665 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c6z6g" Jul 16 12:46:50.008997 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:46:50.008969 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd1c8e4_231a_4a42_88c3_9f5d8c653f7b.slice/crio-abf733a8f2926f43af6a98232af85ea5c1233ae7c1d289443f925809e0fe581c WatchSource:0}: Error finding container abf733a8f2926f43af6a98232af85ea5c1233ae7c1d289443f925809e0fe581c: Status 404 returned error can't find the container with id abf733a8f2926f43af6a98232af85ea5c1233ae7c1d289443f925809e0fe581c Jul 16 12:46:50.102274 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:50.102209 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c6z6g" event={"ID":"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b","Type":"ContainerStarted","Data":"abf733a8f2926f43af6a98232af85ea5c1233ae7c1d289443f925809e0fe581c"} Jul 16 12:46:50.135712 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:50.135647 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb"] Jul 16 12:46:50.139205 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:46:50.139177 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5280694_1df9_4c77_ac14_eee54b7cc27f.slice/crio-6894a45916d5c3482d393cb12f3577506cc9b77200d3c05c905ac42f8bafecd8 WatchSource:0}: Error finding container 6894a45916d5c3482d393cb12f3577506cc9b77200d3c05c905ac42f8bafecd8: Status 404 returned error can't find the container with id 6894a45916d5c3482d393cb12f3577506cc9b77200d3c05c905ac42f8bafecd8 Jul 16 12:46:50.388149 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:50.388129 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:46:51.107182 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:51.107146 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" event={"ID":"c5280694-1df9-4c77-ac14-eee54b7cc27f","Type":"ContainerStarted","Data":"1cae3449561b5bc5ba0340411605c47c3bf25808dd177e2652af2cd372eead29"} Jul 16 12:46:51.107444 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:51.107203 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" event={"ID":"c5280694-1df9-4c77-ac14-eee54b7cc27f","Type":"ContainerStarted","Data":"90fd1c4ab721a4a8582b4f8d031a3529fcd2159beaec0c1b18eecc00089fd3c0"} Jul 16 12:46:51.107444 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:51.107243 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" event={"ID":"c5280694-1df9-4c77-ac14-eee54b7cc27f","Type":"ContainerStarted","Data":"6894a45916d5c3482d393cb12f3577506cc9b77200d3c05c905ac42f8bafecd8"} Jul 16 12:46:51.108932 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:51.108903 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxrcl" event={"ID":"0031d292-78ee-4ea2-b31f-adb988b6d5e3","Type":"ContainerStarted","Data":"53302473b8c397ed20f613a8fa4ff8773ac9d991fabb2f42992b35b56c977335"} Jul 16 12:46:52.114265 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.114208 2565 generic.go:356] "Generic (PLEG): container finished" podID="3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b" containerID="699c00743ee997a30763c47e591e3d175e2d92ee1525e324956f9434a05cff53" exitCode=0 Jul 16 12:46:52.114659 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.114303 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c6z6g" event={"ID":"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b","Type":"ContainerDied","Data":"699c00743ee997a30763c47e591e3d175e2d92ee1525e324956f9434a05cff53"} Jul 16 12:46:52.551092 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.551067 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f4965c6b-jppmp"] Jul 16 12:46:52.564435 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.564411 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.568046 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.568023 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 16 12:46:52.568357 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.568292 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 16 12:46:52.568849 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.568814 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 12:46:52.568939 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.568877 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-d7shg\"" Jul 16 12:46:52.569063 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.569046 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 16 12:46:52.569181 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.569129 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7oim0o5opfeo9\"" Jul 16 12:46:52.571548 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.571318 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f4965c6b-jppmp"] Jul 16 12:46:52.655049 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.654974 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcrrx\" (UniqueName: \"kubernetes.io/projected/80882f06-3d35-44aa-8629-f40dc451f7aa-kube-api-access-jcrrx\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.655049 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.655020 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80882f06-3d35-44aa-8629-f40dc451f7aa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.655263 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.655071 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/80882f06-3d35-44aa-8629-f40dc451f7aa-audit-log\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.655263 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.655120 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/80882f06-3d35-44aa-8629-f40dc451f7aa-client-ca-bundle\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.655263 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.655203 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/80882f06-3d35-44aa-8629-f40dc451f7aa-metrics-server-audit-profiles\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.655389 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.655308 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/80882f06-3d35-44aa-8629-f40dc451f7aa-secret-metrics-server-client-certs\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.655389 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.655344 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/80882f06-3d35-44aa-8629-f40dc451f7aa-secret-metrics-server-tls\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.723212 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.723178 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv"] Jul 16 12:46:52.744120 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.744090 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv"] Jul 16 12:46:52.744297 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.744210 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv" Jul 16 12:46:52.747055 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.746801 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 16 12:46:52.747055 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.746888 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v4f6p\"" Jul 16 12:46:52.756944 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.756326 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/80882f06-3d35-44aa-8629-f40dc451f7aa-client-ca-bundle\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.756944 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.756373 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/80882f06-3d35-44aa-8629-f40dc451f7aa-metrics-server-audit-profiles\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.756944 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.756418 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/80882f06-3d35-44aa-8629-f40dc451f7aa-secret-metrics-server-client-certs\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.756944 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.756451 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/80882f06-3d35-44aa-8629-f40dc451f7aa-secret-metrics-server-tls\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.756944 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.756484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcrrx\" (UniqueName: \"kubernetes.io/projected/80882f06-3d35-44aa-8629-f40dc451f7aa-kube-api-access-jcrrx\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.756944 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.756508 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80882f06-3d35-44aa-8629-f40dc451f7aa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.756944 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.756559 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/80882f06-3d35-44aa-8629-f40dc451f7aa-audit-log\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.756944 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.756860 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/80882f06-3d35-44aa-8629-f40dc451f7aa-audit-log\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.757872 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.757731 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80882f06-3d35-44aa-8629-f40dc451f7aa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.758698 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.758666 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/80882f06-3d35-44aa-8629-f40dc451f7aa-metrics-server-audit-profiles\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.759750 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.759726 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/80882f06-3d35-44aa-8629-f40dc451f7aa-secret-metrics-server-tls\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.759876 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.759848 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/80882f06-3d35-44aa-8629-f40dc451f7aa-secret-metrics-server-client-certs\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.760962 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.760939 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/80882f06-3d35-44aa-8629-f40dc451f7aa-client-ca-bundle\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.771695 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.771674 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcrrx\" (UniqueName: \"kubernetes.io/projected/80882f06-3d35-44aa-8629-f40dc451f7aa-kube-api-access-jcrrx\") pod \"metrics-server-5f4965c6b-jppmp\" (UID: \"80882f06-3d35-44aa-8629-f40dc451f7aa\") " pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.857649 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.857598 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2cb7eec3-02a8-4a53-ac8d-af061fde4a9c-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-xvmgv\" (UID: \"2cb7eec3-02a8-4a53-ac8d-af061fde4a9c\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv" Jul 16 12:46:52.857836 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.857670 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b59ht\" (UniqueName: \"kubernetes.io/projected/2cb7eec3-02a8-4a53-ac8d-af061fde4a9c-kube-api-access-b59ht\") pod \"monitoring-plugin-5f68df7f58-xvmgv\" (UID: \"2cb7eec3-02a8-4a53-ac8d-af061fde4a9c\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv" Jul 16 12:46:52.878203 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.878169 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:46:52.958359 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.958321 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2cb7eec3-02a8-4a53-ac8d-af061fde4a9c-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-xvmgv\" (UID: \"2cb7eec3-02a8-4a53-ac8d-af061fde4a9c\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv" Jul 16 12:46:52.958534 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:52.958414 2565 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 16 12:46:52.958534 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.958409 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b59ht\" (UniqueName: \"kubernetes.io/projected/2cb7eec3-02a8-4a53-ac8d-af061fde4a9c-kube-api-access-b59ht\") pod \"monitoring-plugin-5f68df7f58-xvmgv\" (UID: \"2cb7eec3-02a8-4a53-ac8d-af061fde4a9c\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv" Jul 16 12:46:52.958534 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:46:52.958482 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb7eec3-02a8-4a53-ac8d-af061fde4a9c-monitoring-plugin-cert podName:2cb7eec3-02a8-4a53-ac8d-af061fde4a9c nodeName:}" failed. No retries permitted until 2026-07-16 12:46:53.458460442 +0000 UTC m=+121.453601190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/2cb7eec3-02a8-4a53-ac8d-af061fde4a9c-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-xvmgv" (UID: "2cb7eec3-02a8-4a53-ac8d-af061fde4a9c") : secret "monitoring-plugin-cert" not found Jul 16 12:46:52.974972 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:52.974887 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b59ht\" (UniqueName: \"kubernetes.io/projected/2cb7eec3-02a8-4a53-ac8d-af061fde4a9c-kube-api-access-b59ht\") pod \"monitoring-plugin-5f68df7f58-xvmgv\" (UID: \"2cb7eec3-02a8-4a53-ac8d-af061fde4a9c\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv" Jul 16 12:46:53.060124 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:53.060089 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f4965c6b-jppmp"] Jul 16 12:46:53.063917 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:46:53.063888 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80882f06_3d35_44aa_8629_f40dc451f7aa.slice/crio-8af7524321fc7fffe1173f2a7f8387f5000a86baf4b69fc3e981785256299321 WatchSource:0}: Error finding container 8af7524321fc7fffe1173f2a7f8387f5000a86baf4b69fc3e981785256299321: Status 404 returned error can't find the container with id 8af7524321fc7fffe1173f2a7f8387f5000a86baf4b69fc3e981785256299321 Jul 16 12:46:53.122708 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:53.122670 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" event={"ID":"c5280694-1df9-4c77-ac14-eee54b7cc27f","Type":"ContainerStarted","Data":"c56712b9d1edbd623269ad0acdcba88465969942d6727222b35b553493558137"} Jul 16 12:46:53.126544 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:53.126516 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c6z6g" event={"ID":"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b","Type":"ContainerStarted","Data":"299a19e7f8e0ef0c0c355425035d7b35e5f510734452e5420ce3ed0565a3e9bd"} Jul 16 12:46:53.126677 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:53.126569 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c6z6g" event={"ID":"3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b","Type":"ContainerStarted","Data":"3b01ddca5325a30206878316235b3ea624441348b6831f2b04f011adb3851adf"} Jul 16 12:46:53.128158 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:53.128129 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" event={"ID":"80882f06-3d35-44aa-8629-f40dc451f7aa","Type":"ContainerStarted","Data":"8af7524321fc7fffe1173f2a7f8387f5000a86baf4b69fc3e981785256299321"} Jul 16 12:46:53.151647 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:53.151593 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p6swb" podStartSLOduration=3.331667005 podStartE2EDuration="5.151577599s" totalImagesPullingTime="1.819910594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:53.150087811 +0000 UTC m=+121.145228584" watchObservedRunningTime="2026-07-16 12:46:53.151577599 +0000 UTC m=+121.146718399" Jul 16 12:46:53.175516 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:53.175413 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c6z6g" podStartSLOduration=3.674841273 podStartE2EDuration="5.175392001s" totalImagesPullingTime="1.388970156s" totalInitContainerRuntime="111.580572ms" isStatefulPod=true podCreationTimestamp="2026-07-16 12:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:53.173676092 +0000 UTC m=+121.168816854" watchObservedRunningTime="2026-07-16 12:46:53.175392001 +0000 UTC m=+121.170532761" Jul 16 12:46:53.464109 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:53.464028 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2cb7eec3-02a8-4a53-ac8d-af061fde4a9c-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-xvmgv\" (UID: \"2cb7eec3-02a8-4a53-ac8d-af061fde4a9c\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv" Jul 16 12:46:53.466727 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:53.466696 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2cb7eec3-02a8-4a53-ac8d-af061fde4a9c-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-xvmgv\" (UID: \"2cb7eec3-02a8-4a53-ac8d-af061fde4a9c\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv" Jul 16 12:46:53.657588 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:53.657554 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv" Jul 16 12:46:53.828734 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:53.828611 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv"] Jul 16 12:46:55.801084 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:46:55.801053 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb7eec3_02a8_4a53_ac8d_af061fde4a9c.slice/crio-48560bd8e00b209fb49759fc9fc3c191665afd65589398c0bcecfad38b3b854e WatchSource:0}: Error finding container 48560bd8e00b209fb49759fc9fc3c191665afd65589398c0bcecfad38b3b854e: Status 404 returned error can't find the container with id 48560bd8e00b209fb49759fc9fc3c191665afd65589398c0bcecfad38b3b854e Jul 16 12:46:56.089967 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:56.089878 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5dxp9" Jul 16 12:46:56.144360 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:56.144321 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv" event={"ID":"2cb7eec3-02a8-4a53-ac8d-af061fde4a9c","Type":"ContainerStarted","Data":"48560bd8e00b209fb49759fc9fc3c191665afd65589398c0bcecfad38b3b854e"} Jul 16 12:46:57.151193 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:57.151157 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxrcl" event={"ID":"0031d292-78ee-4ea2-b31f-adb988b6d5e3","Type":"ContainerStarted","Data":"f17873693ae80430b2f06a01d56e9cd26f21f58a32a3a8426758b4a245dfa1c4"} Jul 16 12:46:57.153105 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:57.153076 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" event={"ID":"80882f06-3d35-44aa-8629-f40dc451f7aa","Type":"ContainerStarted","Data":"d94e308baaa21a00cbb95366db926722f6c0f6fa853305cf1039513195a6efab"} Jul 16 12:46:57.177103 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:57.177044 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hxrcl" podStartSLOduration=40.91209649 podStartE2EDuration="1m2.177022206s" totalImagesPullingTime="21.264925716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:45:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:57.174810588 +0000 UTC m=+125.169951347" watchObservedRunningTime="2026-07-16 12:46:57.177022206 +0000 UTC m=+125.172162967" Jul 16 12:46:57.201191 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:57.200294 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" podStartSLOduration=1.591316603 podStartE2EDuration="5.200271617s" totalImagesPullingTime="3.608955014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:57.199735008 +0000 UTC m=+125.194875768" watchObservedRunningTime="2026-07-16 12:46:57.200271617 +0000 UTC m=+125.195412374" Jul 16 12:46:58.158956 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:58.158912 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv" event={"ID":"2cb7eec3-02a8-4a53-ac8d-af061fde4a9c","Type":"ContainerStarted","Data":"83366f630970f4fc562e66da86d094c7303c09084ab1ff589ace1fdea8d1ef5f"} Jul 16 12:46:58.159421 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:58.159353 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv" Jul 16 12:46:58.163850 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:58.163827 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv" Jul 16 12:46:58.177682 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:46:58.177644 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xvmgv" podStartSLOduration=4.21787833 podStartE2EDuration="6.177630084s" totalImagesPullingTime="1.959751754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:58.176737896 +0000 UTC m=+126.171878674" watchObservedRunningTime="2026-07-16 12:46:58.177630084 +0000 UTC m=+126.172770844" Jul 16 12:47:01.339850 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:01.339809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs\") pod \"network-metrics-daemon-n2c94\" (UID: \"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe\") " pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:47:01.342192 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:01.342164 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3f2e5f-9499-4947-8d80-160b3ffd0bbe-metrics-certs\") pod \"network-metrics-daemon-n2c94\" (UID: \"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe\") " pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:47:01.555799 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:01.555766 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xrlr5\"" Jul 16 12:47:01.564178 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:01.564150 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2c94" Jul 16 12:47:01.716092 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:01.716061 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n2c94"] Jul 16 12:47:01.719339 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:47:01.719298 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3f2e5f_9499_4947_8d80_160b3ffd0bbe.slice/crio-139997d02311eff0442039eeeecbb661ff40a032f2a9e49924b66e1a0e251155 WatchSource:0}: Error finding container 139997d02311eff0442039eeeecbb661ff40a032f2a9e49924b66e1a0e251155: Status 404 returned error can't find the container with id 139997d02311eff0442039eeeecbb661ff40a032f2a9e49924b66e1a0e251155 Jul 16 12:47:02.178846 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:02.178799 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n2c94" event={"ID":"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe","Type":"ContainerStarted","Data":"139997d02311eff0442039eeeecbb661ff40a032f2a9e49924b66e1a0e251155"} Jul 16 12:47:04.191182 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:04.191142 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n2c94" event={"ID":"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe","Type":"ContainerStarted","Data":"1eb463585e2c9671f8f52a8cae9bce8c5933cae78d707ae5631649afafbed51a"} Jul 16 12:47:04.191670 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:04.191194 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n2c94" event={"ID":"dd3f2e5f-9499-4947-8d80-160b3ffd0bbe","Type":"ContainerStarted","Data":"4c96ef25d138ab127939eac2a29d1f1fa34ccdccb02694f13ec8db9c56a07768"} Jul 16 12:47:04.210742 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:04.210690 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n2c94" podStartSLOduration=130.812764055 podStartE2EDuration="2m12.210675196s" totalImagesPullingTime="1.397911141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:47:04.210190237 +0000 UTC m=+132.205330997" watchObservedRunningTime="2026-07-16 12:47:04.210675196 +0000 UTC m=+132.205815955" Jul 16 12:47:05.196820 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.196784 2565 generic.go:356] "Generic (PLEG): container finished" podID="5e27e160-a28d-4523-a137-fb415f779ba5" containerID="ebf1a86ad40cbec3f17acd457ecac2d980ba221d7986b66ba1041e94e6de1326" exitCode=0 Jul 16 12:47:05.197273 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.196860 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc" event={"ID":"5e27e160-a28d-4523-a137-fb415f779ba5","Type":"ContainerDied","Data":"ebf1a86ad40cbec3f17acd457ecac2d980ba221d7986b66ba1041e94e6de1326"} Jul 16 12:47:05.197395 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.197380 2565 scope.go:117] "RemoveContainer" containerID="ebf1a86ad40cbec3f17acd457ecac2d980ba221d7986b66ba1041e94e6de1326" Jul 16 12:47:05.413770 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.413715 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" podUID="9eb17f11-3a02-4674-b023-7b781b393e38" containerName="registry" containerID="cri-o://50462a6d2a2961156d457de71e6a6c7eb00402b9329bf5fd7d691a18be77ce56" gracePeriod=30 Jul 16 12:47:05.697777 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.697747 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:47:05.778246 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.778199 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9eb17f11-3a02-4674-b023-7b781b393e38-ca-trust-extracted\") pod \"9eb17f11-3a02-4674-b023-7b781b393e38\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " Jul 16 12:47:05.778444 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.778264 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q9p4\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-kube-api-access-9q9p4\") pod \"9eb17f11-3a02-4674-b023-7b781b393e38\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " Jul 16 12:47:05.778444 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.778290 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-bound-sa-token\") pod \"9eb17f11-3a02-4674-b023-7b781b393e38\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " Jul 16 12:47:05.778444 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.778330 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls\") pod \"9eb17f11-3a02-4674-b023-7b781b393e38\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " Jul 16 12:47:05.778444 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.778371 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9eb17f11-3a02-4674-b023-7b781b393e38-trusted-ca\") pod \"9eb17f11-3a02-4674-b023-7b781b393e38\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " Jul 16 12:47:05.778444 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.778404 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9eb17f11-3a02-4674-b023-7b781b393e38-registry-certificates\") pod \"9eb17f11-3a02-4674-b023-7b781b393e38\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " Jul 16 12:47:05.778709 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.778462 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9eb17f11-3a02-4674-b023-7b781b393e38-image-registry-private-configuration\") pod \"9eb17f11-3a02-4674-b023-7b781b393e38\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " Jul 16 12:47:05.778709 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.778497 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9eb17f11-3a02-4674-b023-7b781b393e38-installation-pull-secrets\") pod \"9eb17f11-3a02-4674-b023-7b781b393e38\" (UID: \"9eb17f11-3a02-4674-b023-7b781b393e38\") " Jul 16 12:47:05.778895 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.778865 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb17f11-3a02-4674-b023-7b781b393e38-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9eb17f11-3a02-4674-b023-7b781b393e38" (UID: "9eb17f11-3a02-4674-b023-7b781b393e38"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:47:05.778958 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.778919 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb17f11-3a02-4674-b023-7b781b393e38-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9eb17f11-3a02-4674-b023-7b781b393e38" (UID: "9eb17f11-3a02-4674-b023-7b781b393e38"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:47:05.781165 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.781133 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-kube-api-access-9q9p4" (OuterVolumeSpecName: "kube-api-access-9q9p4") pod "9eb17f11-3a02-4674-b023-7b781b393e38" (UID: "9eb17f11-3a02-4674-b023-7b781b393e38"). InnerVolumeSpecName "kube-api-access-9q9p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:47:05.781403 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.781363 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9eb17f11-3a02-4674-b023-7b781b393e38" (UID: "9eb17f11-3a02-4674-b023-7b781b393e38"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:47:05.781403 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.781383 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9eb17f11-3a02-4674-b023-7b781b393e38" (UID: "9eb17f11-3a02-4674-b023-7b781b393e38"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:47:05.781403 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.781407 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb17f11-3a02-4674-b023-7b781b393e38-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9eb17f11-3a02-4674-b023-7b781b393e38" (UID: "9eb17f11-3a02-4674-b023-7b781b393e38"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:47:05.781649 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.781512 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb17f11-3a02-4674-b023-7b781b393e38-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9eb17f11-3a02-4674-b023-7b781b393e38" (UID: "9eb17f11-3a02-4674-b023-7b781b393e38"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:47:05.787411 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.787388 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb17f11-3a02-4674-b023-7b781b393e38-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9eb17f11-3a02-4674-b023-7b781b393e38" (UID: "9eb17f11-3a02-4674-b023-7b781b393e38"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:47:05.880095 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.880060 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9eb17f11-3a02-4674-b023-7b781b393e38-image-registry-private-configuration\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:47:05.880095 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.880091 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9eb17f11-3a02-4674-b023-7b781b393e38-installation-pull-secrets\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:47:05.880312 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.880103 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9eb17f11-3a02-4674-b023-7b781b393e38-ca-trust-extracted\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:47:05.880312 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.880116 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q9p4\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-kube-api-access-9q9p4\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:47:05.880312 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.880127 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-bound-sa-token\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:47:05.880312 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.880138 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9eb17f11-3a02-4674-b023-7b781b393e38-registry-tls\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:47:05.880312 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.880151 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9eb17f11-3a02-4674-b023-7b781b393e38-trusted-ca\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:47:05.880312 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:05.880163 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9eb17f11-3a02-4674-b023-7b781b393e38-registry-certificates\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:47:06.201864 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:06.201772 2565 generic.go:356] "Generic (PLEG): container finished" podID="9eb17f11-3a02-4674-b023-7b781b393e38" containerID="50462a6d2a2961156d457de71e6a6c7eb00402b9329bf5fd7d691a18be77ce56" exitCode=0 Jul 16 12:47:06.201864 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:06.201834 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" Jul 16 12:47:06.202459 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:06.201854 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" event={"ID":"9eb17f11-3a02-4674-b023-7b781b393e38","Type":"ContainerDied","Data":"50462a6d2a2961156d457de71e6a6c7eb00402b9329bf5fd7d691a18be77ce56"} Jul 16 12:47:06.202459 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:06.201936 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55c4f54f86-chtl8" event={"ID":"9eb17f11-3a02-4674-b023-7b781b393e38","Type":"ContainerDied","Data":"07b15dd64d5b5dffb4113d4e32fe86317fb99632be546990158eb14f5d711e09"} Jul 16 12:47:06.202459 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:06.201966 2565 scope.go:117] "RemoveContainer" containerID="50462a6d2a2961156d457de71e6a6c7eb00402b9329bf5fd7d691a18be77ce56" Jul 16 12:47:06.203840 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:06.203820 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-brblc" event={"ID":"5e27e160-a28d-4523-a137-fb415f779ba5","Type":"ContainerStarted","Data":"25dad877b8b4bf0fa31c70a354780af31f4b009155273bae8b426a360f2861ae"} Jul 16 12:47:06.211927 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:06.211907 2565 scope.go:117] "RemoveContainer" containerID="50462a6d2a2961156d457de71e6a6c7eb00402b9329bf5fd7d691a18be77ce56" Jul 16 12:47:06.212169 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:47:06.212151 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50462a6d2a2961156d457de71e6a6c7eb00402b9329bf5fd7d691a18be77ce56\": container with ID starting with 50462a6d2a2961156d457de71e6a6c7eb00402b9329bf5fd7d691a18be77ce56 not found: ID does not exist" containerID="50462a6d2a2961156d457de71e6a6c7eb00402b9329bf5fd7d691a18be77ce56" Jul 16 12:47:06.212223 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:06.212176 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50462a6d2a2961156d457de71e6a6c7eb00402b9329bf5fd7d691a18be77ce56"} err="failed to get container status \"50462a6d2a2961156d457de71e6a6c7eb00402b9329bf5fd7d691a18be77ce56\": rpc error: code = NotFound desc = could not find container \"50462a6d2a2961156d457de71e6a6c7eb00402b9329bf5fd7d691a18be77ce56\": container with ID starting with 50462a6d2a2961156d457de71e6a6c7eb00402b9329bf5fd7d691a18be77ce56 not found: ID does not exist" Jul 16 12:47:06.242497 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:06.242464 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55c4f54f86-chtl8"] Jul 16 12:47:06.256056 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:06.256027 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55c4f54f86-chtl8"] Jul 16 12:47:06.544071 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:06.544039 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eb17f11-3a02-4674-b023-7b781b393e38" path="/var/lib/kubelet/pods/9eb17f11-3a02-4674-b023-7b781b393e38/volumes" Jul 16 12:47:12.878997 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:12.878960 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:47:12.879470 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:12.879022 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:47:14.116860 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.116820 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57457978f4-hgqb2" podUID="fe654dab-4eec-4c27-be43-5fe243225ef9" containerName="console" containerID="cri-o://17e1b62fa1c646488eecf1b0780f9fca5baf75efb8cc2ce74ee4b508e5a45432" gracePeriod=15 Jul 16 12:47:14.237501 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.237467 2565 generic.go:356] "Generic (PLEG): container finished" podID="fe654dab-4eec-4c27-be43-5fe243225ef9" containerID="17e1b62fa1c646488eecf1b0780f9fca5baf75efb8cc2ce74ee4b508e5a45432" exitCode=0 Jul 16 12:47:14.237635 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.237546 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57457978f4-hgqb2" event={"ID":"fe654dab-4eec-4c27-be43-5fe243225ef9","Type":"ContainerDied","Data":"17e1b62fa1c646488eecf1b0780f9fca5baf75efb8cc2ce74ee4b508e5a45432"} Jul 16 12:47:14.357150 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.357119 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:47:14.444306 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.444166 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-oauth-serving-cert\") pod \"fe654dab-4eec-4c27-be43-5fe243225ef9\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " Jul 16 12:47:14.444306 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.444207 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe654dab-4eec-4c27-be43-5fe243225ef9-console-oauth-config\") pod \"fe654dab-4eec-4c27-be43-5fe243225ef9\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " Jul 16 12:47:14.444306 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.444258 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe654dab-4eec-4c27-be43-5fe243225ef9-console-serving-cert\") pod \"fe654dab-4eec-4c27-be43-5fe243225ef9\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " Jul 16 12:47:14.444306 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.444281 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvp8g\" (UniqueName: \"kubernetes.io/projected/fe654dab-4eec-4c27-be43-5fe243225ef9-kube-api-access-gvp8g\") pod \"fe654dab-4eec-4c27-be43-5fe243225ef9\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " Jul 16 12:47:14.444629 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.444330 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-service-ca\") pod \"fe654dab-4eec-4c27-be43-5fe243225ef9\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " Jul 16 12:47:14.444629 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.444393 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-console-config\") pod \"fe654dab-4eec-4c27-be43-5fe243225ef9\" (UID: \"fe654dab-4eec-4c27-be43-5fe243225ef9\") " Jul 16 12:47:14.444629 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.444566 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fe654dab-4eec-4c27-be43-5fe243225ef9" (UID: "fe654dab-4eec-4c27-be43-5fe243225ef9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:47:14.444750 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.444738 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-service-ca" (OuterVolumeSpecName: "service-ca") pod "fe654dab-4eec-4c27-be43-5fe243225ef9" (UID: "fe654dab-4eec-4c27-be43-5fe243225ef9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:47:14.444860 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.444834 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-console-config" (OuterVolumeSpecName: "console-config") pod "fe654dab-4eec-4c27-be43-5fe243225ef9" (UID: "fe654dab-4eec-4c27-be43-5fe243225ef9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:47:14.446635 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.446607 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe654dab-4eec-4c27-be43-5fe243225ef9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fe654dab-4eec-4c27-be43-5fe243225ef9" (UID: "fe654dab-4eec-4c27-be43-5fe243225ef9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:47:14.446980 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.446963 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe654dab-4eec-4c27-be43-5fe243225ef9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fe654dab-4eec-4c27-be43-5fe243225ef9" (UID: "fe654dab-4eec-4c27-be43-5fe243225ef9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:47:14.447049 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.446992 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe654dab-4eec-4c27-be43-5fe243225ef9-kube-api-access-gvp8g" (OuterVolumeSpecName: "kube-api-access-gvp8g") pod "fe654dab-4eec-4c27-be43-5fe243225ef9" (UID: "fe654dab-4eec-4c27-be43-5fe243225ef9"). InnerVolumeSpecName "kube-api-access-gvp8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:47:14.545979 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.545943 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-console-config\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:47:14.545979 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.545978 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-oauth-serving-cert\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:47:14.546167 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.545995 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe654dab-4eec-4c27-be43-5fe243225ef9-console-oauth-config\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:47:14.546167 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.546009 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe654dab-4eec-4c27-be43-5fe243225ef9-console-serving-cert\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:47:14.546167 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.546023 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvp8g\" (UniqueName: \"kubernetes.io/projected/fe654dab-4eec-4c27-be43-5fe243225ef9-kube-api-access-gvp8g\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:47:14.546167 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:14.546036 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe654dab-4eec-4c27-be43-5fe243225ef9-service-ca\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:47:15.242486 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:15.242456 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57457978f4-hgqb2" Jul 16 12:47:15.242486 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:15.242467 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57457978f4-hgqb2" event={"ID":"fe654dab-4eec-4c27-be43-5fe243225ef9","Type":"ContainerDied","Data":"c2985160fe7b5ca756ee50fc15a8820b654aa03a40f468ea4ded1a46bd7e868e"} Jul 16 12:47:15.242997 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:15.242523 2565 scope.go:117] "RemoveContainer" containerID="17e1b62fa1c646488eecf1b0780f9fca5baf75efb8cc2ce74ee4b508e5a45432" Jul 16 12:47:15.264102 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:15.264070 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57457978f4-hgqb2"] Jul 16 12:47:15.272638 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:15.272611 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57457978f4-hgqb2"] Jul 16 12:47:16.543993 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:16.543958 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe654dab-4eec-4c27-be43-5fe243225ef9" path="/var/lib/kubelet/pods/fe654dab-4eec-4c27-be43-5fe243225ef9/volumes" Jul 16 12:47:25.284247 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:25.284206 2565 generic.go:356] "Generic (PLEG): container finished" podID="73045a2a-1f69-48c1-96f1-54b941192680" containerID="a815cfee93855fdc6608884a05550067a5612ea44dfd7d417fe5f1059e61aeae" exitCode=0 Jul 16 12:47:25.284668 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:25.284265 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" event={"ID":"73045a2a-1f69-48c1-96f1-54b941192680","Type":"ContainerDied","Data":"a815cfee93855fdc6608884a05550067a5612ea44dfd7d417fe5f1059e61aeae"} Jul 16 12:47:25.284668 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:25.284633 2565 scope.go:117] "RemoveContainer" containerID="a815cfee93855fdc6608884a05550067a5612ea44dfd7d417fe5f1059e61aeae" Jul 16 12:47:26.292437 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:26.292397 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qbhg8" event={"ID":"73045a2a-1f69-48c1-96f1-54b941192680","Type":"ContainerStarted","Data":"9232084dfa67fb4b0cf1da3f488f5ab6bd1357bc13ebe69133f41ef729090a94"} Jul 16 12:47:32.883772 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:32.883742 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:47:32.887749 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:47:32.887727 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f4965c6b-jppmp" Jul 16 12:48:10.520061 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.520030 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9bb975cf7-qgz42"] Jul 16 12:48:10.520529 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.520336 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb17f11-3a02-4674-b023-7b781b393e38" containerName="registry" Jul 16 12:48:10.520529 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.520348 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb17f11-3a02-4674-b023-7b781b393e38" containerName="registry" Jul 16 12:48:10.520529 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.520356 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe654dab-4eec-4c27-be43-5fe243225ef9" containerName="console" Jul 16 12:48:10.520529 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.520362 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe654dab-4eec-4c27-be43-5fe243225ef9" containerName="console" Jul 16 12:48:10.520529 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.520409 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eb17f11-3a02-4674-b023-7b781b393e38" containerName="registry" Jul 16 12:48:10.520529 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.520415 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe654dab-4eec-4c27-be43-5fe243225ef9" containerName="console" Jul 16 12:48:10.522893 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.522877 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.529806 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.529786 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 12:48:10.529913 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.529786 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 12:48:10.530912 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.530892 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 12:48:10.531027 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.530916 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 12:48:10.531027 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.530919 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 12:48:10.531027 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.530899 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-678j9\"" Jul 16 12:48:10.534578 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.534561 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 12:48:10.553943 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.553920 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9bb975cf7-qgz42"] Jul 16 12:48:10.571898 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.571871 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-trusted-ca-bundle\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.572012 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.571901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m24k\" (UniqueName: \"kubernetes.io/projected/e883afe2-3f49-48ab-9759-974238eb7f69-kube-api-access-5m24k\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.572012 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.571926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-console-config\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.572012 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.571977 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-service-ca\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.572012 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.572004 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e883afe2-3f49-48ab-9759-974238eb7f69-console-serving-cert\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.572167 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.572047 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e883afe2-3f49-48ab-9759-974238eb7f69-console-oauth-config\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.572167 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.572062 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-oauth-serving-cert\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.672402 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.672366 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-trusted-ca-bundle\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.672402 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.672401 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m24k\" (UniqueName: \"kubernetes.io/projected/e883afe2-3f49-48ab-9759-974238eb7f69-kube-api-access-5m24k\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.672639 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.672422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-console-config\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.672639 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.672521 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-service-ca\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.672639 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.672570 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e883afe2-3f49-48ab-9759-974238eb7f69-console-serving-cert\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.672639 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.672633 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e883afe2-3f49-48ab-9759-974238eb7f69-console-oauth-config\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.672817 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.672658 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-oauth-serving-cert\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.673187 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.673154 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-console-config\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.673337 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.673295 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-trusted-ca-bundle\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.673337 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.673315 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-service-ca\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.673419 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.673354 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-oauth-serving-cert\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.675614 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.675586 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e883afe2-3f49-48ab-9759-974238eb7f69-console-oauth-config\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.675817 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.675797 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e883afe2-3f49-48ab-9759-974238eb7f69-console-serving-cert\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.685284 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.685214 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m24k\" (UniqueName: \"kubernetes.io/projected/e883afe2-3f49-48ab-9759-974238eb7f69-kube-api-access-5m24k\") pod \"console-9bb975cf7-qgz42\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.832722 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.832628 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:10.969987 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:10.969963 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9bb975cf7-qgz42"] Jul 16 12:48:10.972427 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:48:10.972397 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode883afe2_3f49_48ab_9759_974238eb7f69.slice/crio-f1ed133704a2c5c6ad8497d1b586a8bf8a02f06167bf856ad42e4724ba7b14fa WatchSource:0}: Error finding container f1ed133704a2c5c6ad8497d1b586a8bf8a02f06167bf856ad42e4724ba7b14fa: Status 404 returned error can't find the container with id f1ed133704a2c5c6ad8497d1b586a8bf8a02f06167bf856ad42e4724ba7b14fa Jul 16 12:48:11.464682 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:11.464647 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bb975cf7-qgz42" event={"ID":"e883afe2-3f49-48ab-9759-974238eb7f69","Type":"ContainerStarted","Data":"533ef659f6f67b02db1de90fbfe39920c958f4c609f96c84852937be39c62a7a"} Jul 16 12:48:11.464830 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:11.464700 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bb975cf7-qgz42" event={"ID":"e883afe2-3f49-48ab-9759-974238eb7f69","Type":"ContainerStarted","Data":"f1ed133704a2c5c6ad8497d1b586a8bf8a02f06167bf856ad42e4724ba7b14fa"} Jul 16 12:48:11.484995 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:11.484944 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9bb975cf7-qgz42" podStartSLOduration=1.484927853 podStartE2EDuration="1.484927853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:48:11.483988056 +0000 UTC m=+199.479128834" watchObservedRunningTime="2026-07-16 12:48:11.484927853 +0000 UTC m=+199.480068613" Jul 16 12:48:20.832987 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:20.832946 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:20.833488 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:20.833024 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:20.837747 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:20.837724 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:48:21.505179 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:48:21.505151 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:49:31.076506 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.076468 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz"] Jul 16 12:49:31.078702 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.078685 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" Jul 16 12:49:31.081421 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.081401 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 12:49:31.082289 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.082271 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 12:49:31.082289 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.082281 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5v8g6\"" Jul 16 12:49:31.095261 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.095216 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz"] Jul 16 12:49:31.101073 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.101050 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0b5ba08-cbac-4589-ba87-d14882819789-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz\" (UID: \"f0b5ba08-cbac-4589-ba87-d14882819789\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" Jul 16 12:49:31.101172 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.101104 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvj86\" (UniqueName: \"kubernetes.io/projected/f0b5ba08-cbac-4589-ba87-d14882819789-kube-api-access-vvj86\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz\" (UID: \"f0b5ba08-cbac-4589-ba87-d14882819789\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" Jul 16 12:49:31.101253 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.101174 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0b5ba08-cbac-4589-ba87-d14882819789-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz\" (UID: \"f0b5ba08-cbac-4589-ba87-d14882819789\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" Jul 16 12:49:31.201798 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.201753 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0b5ba08-cbac-4589-ba87-d14882819789-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz\" (UID: \"f0b5ba08-cbac-4589-ba87-d14882819789\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" Jul 16 12:49:31.201998 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.201822 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0b5ba08-cbac-4589-ba87-d14882819789-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz\" (UID: \"f0b5ba08-cbac-4589-ba87-d14882819789\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" Jul 16 12:49:31.201998 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.201878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvj86\" (UniqueName: \"kubernetes.io/projected/f0b5ba08-cbac-4589-ba87-d14882819789-kube-api-access-vvj86\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz\" (UID: \"f0b5ba08-cbac-4589-ba87-d14882819789\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" Jul 16 12:49:31.202175 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.202151 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0b5ba08-cbac-4589-ba87-d14882819789-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz\" (UID: \"f0b5ba08-cbac-4589-ba87-d14882819789\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" Jul 16 12:49:31.202270 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.202182 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0b5ba08-cbac-4589-ba87-d14882819789-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz\" (UID: \"f0b5ba08-cbac-4589-ba87-d14882819789\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" Jul 16 12:49:31.212285 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.212257 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvj86\" (UniqueName: \"kubernetes.io/projected/f0b5ba08-cbac-4589-ba87-d14882819789-kube-api-access-vvj86\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz\" (UID: \"f0b5ba08-cbac-4589-ba87-d14882819789\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" Jul 16 12:49:31.387753 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.387660 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" Jul 16 12:49:31.523957 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.523932 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz"] Jul 16 12:49:31.526597 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:49:31.526572 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b5ba08_cbac_4589_ba87_d14882819789.slice/crio-cd8c335fdde4dc7cca99b20ec013c262ca9e47f026f6839c26d3401ee6c09984 WatchSource:0}: Error finding container cd8c335fdde4dc7cca99b20ec013c262ca9e47f026f6839c26d3401ee6c09984: Status 404 returned error can't find the container with id cd8c335fdde4dc7cca99b20ec013c262ca9e47f026f6839c26d3401ee6c09984 Jul 16 12:49:31.761094 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:31.761004 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" event={"ID":"f0b5ba08-cbac-4589-ba87-d14882819789","Type":"ContainerStarted","Data":"cd8c335fdde4dc7cca99b20ec013c262ca9e47f026f6839c26d3401ee6c09984"} Jul 16 12:49:38.790462 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:38.790426 2565 generic.go:356] "Generic (PLEG): container finished" podID="f0b5ba08-cbac-4589-ba87-d14882819789" containerID="e58a3019c5d9dfc5aa481c49d47f4c36341f59b9b28492ce93368af939c4c849" exitCode=0 Jul 16 12:49:38.790871 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:38.790479 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" event={"ID":"f0b5ba08-cbac-4589-ba87-d14882819789","Type":"ContainerDied","Data":"e58a3019c5d9dfc5aa481c49d47f4c36341f59b9b28492ce93368af939c4c849"} Jul 16 12:49:40.802164 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:40.802131 2565 generic.go:356] "Generic (PLEG): container finished" podID="f0b5ba08-cbac-4589-ba87-d14882819789" containerID="7fa0f70a9532ec60ef74695d04f929b3ce8c728d818764d4af4b4a4b9005cb68" exitCode=0 Jul 16 12:49:40.802529 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:40.802212 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" event={"ID":"f0b5ba08-cbac-4589-ba87-d14882819789","Type":"ContainerDied","Data":"7fa0f70a9532ec60ef74695d04f929b3ce8c728d818764d4af4b4a4b9005cb68"} Jul 16 12:49:46.831554 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:46.831520 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" event={"ID":"f0b5ba08-cbac-4589-ba87-d14882819789","Type":"ContainerStarted","Data":"ba1febe6b82f0b0a32139176a8a3e0396617503fff32dde1bc245fb5ec24cbea"} Jul 16 12:49:46.860325 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:46.860265 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" podStartSLOduration=2.14418574 podStartE2EDuration="15.860244746s" totalImagesPullingTime="13.716059006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:49:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:49:46.858359829 +0000 UTC m=+294.853500590" watchObservedRunningTime="2026-07-16 12:49:46.860244746 +0000 UTC m=+294.855385497" Jul 16 12:49:47.837347 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:47.837316 2565 generic.go:356] "Generic (PLEG): container finished" podID="f0b5ba08-cbac-4589-ba87-d14882819789" containerID="ba1febe6b82f0b0a32139176a8a3e0396617503fff32dde1bc245fb5ec24cbea" exitCode=0 Jul 16 12:49:47.837705 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:47.837382 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" event={"ID":"f0b5ba08-cbac-4589-ba87-d14882819789","Type":"ContainerDied","Data":"ba1febe6b82f0b0a32139176a8a3e0396617503fff32dde1bc245fb5ec24cbea"} Jul 16 12:49:48.964162 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:48.964136 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" Jul 16 12:49:49.063225 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:49.063195 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0b5ba08-cbac-4589-ba87-d14882819789-util\") pod \"f0b5ba08-cbac-4589-ba87-d14882819789\" (UID: \"f0b5ba08-cbac-4589-ba87-d14882819789\") " Jul 16 12:49:49.063404 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:49.063267 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvj86\" (UniqueName: \"kubernetes.io/projected/f0b5ba08-cbac-4589-ba87-d14882819789-kube-api-access-vvj86\") pod \"f0b5ba08-cbac-4589-ba87-d14882819789\" (UID: \"f0b5ba08-cbac-4589-ba87-d14882819789\") " Jul 16 12:49:49.063404 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:49.063303 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0b5ba08-cbac-4589-ba87-d14882819789-bundle\") pod \"f0b5ba08-cbac-4589-ba87-d14882819789\" (UID: \"f0b5ba08-cbac-4589-ba87-d14882819789\") " Jul 16 12:49:49.063957 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:49.063931 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b5ba08-cbac-4589-ba87-d14882819789-bundle" (OuterVolumeSpecName: "bundle") pod "f0b5ba08-cbac-4589-ba87-d14882819789" (UID: "f0b5ba08-cbac-4589-ba87-d14882819789"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:49:49.065579 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:49.065555 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b5ba08-cbac-4589-ba87-d14882819789-kube-api-access-vvj86" (OuterVolumeSpecName: "kube-api-access-vvj86") pod "f0b5ba08-cbac-4589-ba87-d14882819789" (UID: "f0b5ba08-cbac-4589-ba87-d14882819789"). InnerVolumeSpecName "kube-api-access-vvj86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:49:49.067399 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:49.067369 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b5ba08-cbac-4589-ba87-d14882819789-util" (OuterVolumeSpecName: "util") pod "f0b5ba08-cbac-4589-ba87-d14882819789" (UID: "f0b5ba08-cbac-4589-ba87-d14882819789"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:49:49.164580 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:49.164482 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvj86\" (UniqueName: \"kubernetes.io/projected/f0b5ba08-cbac-4589-ba87-d14882819789-kube-api-access-vvj86\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:49:49.164580 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:49.164525 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0b5ba08-cbac-4589-ba87-d14882819789-bundle\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:49:49.164580 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:49.164535 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0b5ba08-cbac-4589-ba87-d14882819789-util\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:49:49.846292 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:49.846256 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" event={"ID":"f0b5ba08-cbac-4589-ba87-d14882819789","Type":"ContainerDied","Data":"cd8c335fdde4dc7cca99b20ec013c262ca9e47f026f6839c26d3401ee6c09984"} Jul 16 12:49:49.846454 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:49.846296 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b77kz" Jul 16 12:49:49.846454 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:49.846303 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd8c335fdde4dc7cca99b20ec013c262ca9e47f026f6839c26d3401ee6c09984" Jul 16 12:49:52.437655 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:52.437628 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 12:49:52.440947 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:52.440923 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 12:49:52.441115 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:52.441098 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 12:49:54.592136 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.592106 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hkhkg"] Jul 16 12:49:54.633024 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.592391 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b5ba08-cbac-4589-ba87-d14882819789" containerName="util" Jul 16 12:49:54.633024 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.592402 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b5ba08-cbac-4589-ba87-d14882819789" containerName="util" Jul 16 12:49:54.633024 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.592417 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b5ba08-cbac-4589-ba87-d14882819789" containerName="pull" Jul 16 12:49:54.633024 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.592422 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b5ba08-cbac-4589-ba87-d14882819789" containerName="pull" Jul 16 12:49:54.633024 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.592432 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b5ba08-cbac-4589-ba87-d14882819789" containerName="extract" Jul 16 12:49:54.633024 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.592437 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b5ba08-cbac-4589-ba87-d14882819789" containerName="extract" Jul 16 12:49:54.633024 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.592484 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0b5ba08-cbac-4589-ba87-d14882819789" containerName="extract" Jul 16 12:49:54.640572 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.640549 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hkhkg"] Jul 16 12:49:54.640667 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.640645 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hkhkg" Jul 16 12:49:54.643312 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.643285 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:49:54.643312 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.643282 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hqsfw\"" Jul 16 12:49:54.643508 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.643444 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 16 12:49:54.812795 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.812761 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvvb\" (UniqueName: \"kubernetes.io/projected/899bb369-7ece-449c-90db-13788d291e0a-kube-api-access-swvvb\") pod \"cert-manager-operator-controller-manager-55c444bf86-hkhkg\" (UID: \"899bb369-7ece-449c-90db-13788d291e0a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hkhkg" Jul 16 12:49:54.812944 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.812808 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/899bb369-7ece-449c-90db-13788d291e0a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hkhkg\" (UID: \"899bb369-7ece-449c-90db-13788d291e0a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hkhkg" Jul 16 12:49:54.913863 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.913795 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvvb\" (UniqueName: \"kubernetes.io/projected/899bb369-7ece-449c-90db-13788d291e0a-kube-api-access-swvvb\") pod \"cert-manager-operator-controller-manager-55c444bf86-hkhkg\" (UID: \"899bb369-7ece-449c-90db-13788d291e0a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hkhkg" Jul 16 12:49:54.913863 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.913838 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/899bb369-7ece-449c-90db-13788d291e0a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hkhkg\" (UID: \"899bb369-7ece-449c-90db-13788d291e0a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hkhkg" Jul 16 12:49:54.914182 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.914166 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/899bb369-7ece-449c-90db-13788d291e0a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hkhkg\" (UID: \"899bb369-7ece-449c-90db-13788d291e0a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hkhkg" Jul 16 12:49:54.934672 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.934640 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvvb\" (UniqueName: \"kubernetes.io/projected/899bb369-7ece-449c-90db-13788d291e0a-kube-api-access-swvvb\") pod \"cert-manager-operator-controller-manager-55c444bf86-hkhkg\" (UID: \"899bb369-7ece-449c-90db-13788d291e0a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hkhkg" Jul 16 12:49:54.949508 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:54.949487 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hkhkg" Jul 16 12:49:55.108446 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:55.108418 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hkhkg"] Jul 16 12:49:55.111018 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:49:55.110993 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899bb369_7ece_449c_90db_13788d291e0a.slice/crio-24fc486205dd0f4a4811ab426d94d21f686dbf3452ba9cd8f4e27158acb4da2d WatchSource:0}: Error finding container 24fc486205dd0f4a4811ab426d94d21f686dbf3452ba9cd8f4e27158acb4da2d: Status 404 returned error can't find the container with id 24fc486205dd0f4a4811ab426d94d21f686dbf3452ba9cd8f4e27158acb4da2d Jul 16 12:49:55.113869 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:55.113853 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:49:55.869175 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:55.869134 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hkhkg" event={"ID":"899bb369-7ece-449c-90db-13788d291e0a","Type":"ContainerStarted","Data":"24fc486205dd0f4a4811ab426d94d21f686dbf3452ba9cd8f4e27158acb4da2d"} Jul 16 12:49:57.880914 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:57.880877 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hkhkg" event={"ID":"899bb369-7ece-449c-90db-13788d291e0a","Type":"ContainerStarted","Data":"4c4d45eb4b6773c7c65e34db8471fb79f9551368904c169510389566df0ee408"} Jul 16 12:49:57.906660 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:57.906608 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hkhkg" podStartSLOduration=1.613973263 podStartE2EDuration="3.906588576s" totalImagesPullingTime="2.292615313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:49:57.904939611 +0000 UTC m=+305.900080378" watchObservedRunningTime="2026-07-16 12:49:57.906588576 +0000 UTC m=+305.901729337" Jul 16 12:49:59.004590 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.004559 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs"] Jul 16 12:49:59.026638 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.026606 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs"] Jul 16 12:49:59.026790 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.026701 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" Jul 16 12:49:59.029411 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.029383 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 12:49:59.030209 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.030190 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5v8g6\"" Jul 16 12:49:59.030329 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.030240 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 12:49:59.149452 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.149431 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a124a31a-c7e9-497a-b695-871976c48fa7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs\" (UID: \"a124a31a-c7e9-497a-b695-871976c48fa7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" Jul 16 12:49:59.149569 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.149463 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a124a31a-c7e9-497a-b695-871976c48fa7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs\" (UID: \"a124a31a-c7e9-497a-b695-871976c48fa7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" Jul 16 12:49:59.149569 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.149483 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b22hx\" (UniqueName: \"kubernetes.io/projected/a124a31a-c7e9-497a-b695-871976c48fa7-kube-api-access-b22hx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs\" (UID: \"a124a31a-c7e9-497a-b695-871976c48fa7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" Jul 16 12:49:59.250356 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.250325 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a124a31a-c7e9-497a-b695-871976c48fa7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs\" (UID: \"a124a31a-c7e9-497a-b695-871976c48fa7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" Jul 16 12:49:59.250462 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.250414 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a124a31a-c7e9-497a-b695-871976c48fa7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs\" (UID: \"a124a31a-c7e9-497a-b695-871976c48fa7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" Jul 16 12:49:59.250462 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.250446 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b22hx\" (UniqueName: \"kubernetes.io/projected/a124a31a-c7e9-497a-b695-871976c48fa7-kube-api-access-b22hx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs\" (UID: \"a124a31a-c7e9-497a-b695-871976c48fa7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" Jul 16 12:49:59.250715 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.250696 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a124a31a-c7e9-497a-b695-871976c48fa7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs\" (UID: \"a124a31a-c7e9-497a-b695-871976c48fa7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" Jul 16 12:49:59.250753 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.250735 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a124a31a-c7e9-497a-b695-871976c48fa7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs\" (UID: \"a124a31a-c7e9-497a-b695-871976c48fa7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" Jul 16 12:49:59.261258 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.261183 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b22hx\" (UniqueName: \"kubernetes.io/projected/a124a31a-c7e9-497a-b695-871976c48fa7-kube-api-access-b22hx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs\" (UID: \"a124a31a-c7e9-497a-b695-871976c48fa7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" Jul 16 12:49:59.335671 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.335646 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" Jul 16 12:49:59.420757 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.420726 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2nf2r"] Jul 16 12:49:59.435843 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.435822 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-2nf2r" Jul 16 12:49:59.439174 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.439141 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2nf2r"] Jul 16 12:49:59.439741 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.439597 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 16 12:49:59.439741 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.439612 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 16 12:49:59.439741 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.439603 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6n48w\"" Jul 16 12:49:59.483192 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.483163 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs"] Jul 16 12:49:59.486036 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:49:59.486003 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda124a31a_c7e9_497a_b695_871976c48fa7.slice/crio-fdba0562ee7f05336ccbf960a6e865cd5478e4dfcc6921e202579cdb4b6dbbc9 WatchSource:0}: Error finding container fdba0562ee7f05336ccbf960a6e865cd5478e4dfcc6921e202579cdb4b6dbbc9: Status 404 returned error can't find the container with id fdba0562ee7f05336ccbf960a6e865cd5478e4dfcc6921e202579cdb4b6dbbc9 Jul 16 12:49:59.553878 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.553852 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkgjt\" (UniqueName: \"kubernetes.io/projected/611dc307-70da-48f4-bb2a-f4697e55cf1c-kube-api-access-vkgjt\") pod \"cert-manager-webhook-6486ffd997-2nf2r\" (UID: \"611dc307-70da-48f4-bb2a-f4697e55cf1c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nf2r" Jul 16 12:49:59.554006 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.553892 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/611dc307-70da-48f4-bb2a-f4697e55cf1c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2nf2r\" (UID: \"611dc307-70da-48f4-bb2a-f4697e55cf1c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nf2r" Jul 16 12:49:59.654481 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.654446 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkgjt\" (UniqueName: \"kubernetes.io/projected/611dc307-70da-48f4-bb2a-f4697e55cf1c-kube-api-access-vkgjt\") pod \"cert-manager-webhook-6486ffd997-2nf2r\" (UID: \"611dc307-70da-48f4-bb2a-f4697e55cf1c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nf2r" Jul 16 12:49:59.654711 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.654670 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/611dc307-70da-48f4-bb2a-f4697e55cf1c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2nf2r\" (UID: \"611dc307-70da-48f4-bb2a-f4697e55cf1c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nf2r" Jul 16 12:49:59.663561 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.663538 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/611dc307-70da-48f4-bb2a-f4697e55cf1c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2nf2r\" (UID: \"611dc307-70da-48f4-bb2a-f4697e55cf1c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nf2r" Jul 16 12:49:59.663670 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.663654 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkgjt\" (UniqueName: \"kubernetes.io/projected/611dc307-70da-48f4-bb2a-f4697e55cf1c-kube-api-access-vkgjt\") pod \"cert-manager-webhook-6486ffd997-2nf2r\" (UID: \"611dc307-70da-48f4-bb2a-f4697e55cf1c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nf2r" Jul 16 12:49:59.757947 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.757921 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-2nf2r" Jul 16 12:49:59.891646 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.891613 2565 generic.go:356] "Generic (PLEG): container finished" podID="a124a31a-c7e9-497a-b695-871976c48fa7" containerID="dcaeb620868f789645fc2ae0373cc193825b13ba143a42f59b5a55363cc1095d" exitCode=0 Jul 16 12:49:59.891802 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.891661 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" event={"ID":"a124a31a-c7e9-497a-b695-871976c48fa7","Type":"ContainerDied","Data":"dcaeb620868f789645fc2ae0373cc193825b13ba143a42f59b5a55363cc1095d"} Jul 16 12:49:59.891802 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.891710 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" event={"ID":"a124a31a-c7e9-497a-b695-871976c48fa7","Type":"ContainerStarted","Data":"fdba0562ee7f05336ccbf960a6e865cd5478e4dfcc6921e202579cdb4b6dbbc9"} Jul 16 12:49:59.900742 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:49:59.900681 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2nf2r"] Jul 16 12:49:59.902999 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:49:59.902978 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod611dc307_70da_48f4_bb2a_f4697e55cf1c.slice/crio-b0438f53f47df304bce36e6bc0ed1bd252b54782a1d083fed0133220de1481b0 WatchSource:0}: Error finding container b0438f53f47df304bce36e6bc0ed1bd252b54782a1d083fed0133220de1481b0: Status 404 returned error can't find the container with id b0438f53f47df304bce36e6bc0ed1bd252b54782a1d083fed0133220de1481b0 Jul 16 12:50:00.897600 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:00.897557 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-2nf2r" event={"ID":"611dc307-70da-48f4-bb2a-f4697e55cf1c","Type":"ContainerStarted","Data":"b0438f53f47df304bce36e6bc0ed1bd252b54782a1d083fed0133220de1481b0"} Jul 16 12:50:01.090688 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:01.090659 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nlj7m"] Jul 16 12:50:01.096448 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:01.095100 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nlj7m" Jul 16 12:50:01.098209 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:01.098182 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mh6b7\"" Jul 16 12:50:01.115944 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:01.115921 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nlj7m"] Jul 16 12:50:01.268066 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:01.268027 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d350342-abc4-4bab-a5f5-56c03b02cac9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nlj7m\" (UID: \"0d350342-abc4-4bab-a5f5-56c03b02cac9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nlj7m" Jul 16 12:50:01.268245 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:01.268078 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l77hx\" (UniqueName: \"kubernetes.io/projected/0d350342-abc4-4bab-a5f5-56c03b02cac9-kube-api-access-l77hx\") pod \"cert-manager-cainjector-559cdd8c87-nlj7m\" (UID: \"0d350342-abc4-4bab-a5f5-56c03b02cac9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nlj7m" Jul 16 12:50:01.368915 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:01.368878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l77hx\" (UniqueName: \"kubernetes.io/projected/0d350342-abc4-4bab-a5f5-56c03b02cac9-kube-api-access-l77hx\") pod \"cert-manager-cainjector-559cdd8c87-nlj7m\" (UID: \"0d350342-abc4-4bab-a5f5-56c03b02cac9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nlj7m" Jul 16 12:50:01.369092 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:01.368959 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d350342-abc4-4bab-a5f5-56c03b02cac9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nlj7m\" (UID: \"0d350342-abc4-4bab-a5f5-56c03b02cac9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nlj7m" Jul 16 12:50:01.385022 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:01.384992 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d350342-abc4-4bab-a5f5-56c03b02cac9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nlj7m\" (UID: \"0d350342-abc4-4bab-a5f5-56c03b02cac9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nlj7m" Jul 16 12:50:01.395808 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:01.395783 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l77hx\" (UniqueName: \"kubernetes.io/projected/0d350342-abc4-4bab-a5f5-56c03b02cac9-kube-api-access-l77hx\") pod \"cert-manager-cainjector-559cdd8c87-nlj7m\" (UID: \"0d350342-abc4-4bab-a5f5-56c03b02cac9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nlj7m" Jul 16 12:50:01.406868 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:01.406846 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nlj7m" Jul 16 12:50:01.574171 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:01.574143 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nlj7m"] Jul 16 12:50:01.577499 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:50:01.577469 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d350342_abc4_4bab_a5f5_56c03b02cac9.slice/crio-7638408e009cf37ede10bf5b561833cdc260dac19ecb3fab63baa7dd5db23c7f WatchSource:0}: Error finding container 7638408e009cf37ede10bf5b561833cdc260dac19ecb3fab63baa7dd5db23c7f: Status 404 returned error can't find the container with id 7638408e009cf37ede10bf5b561833cdc260dac19ecb3fab63baa7dd5db23c7f Jul 16 12:50:01.906113 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:01.906012 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nlj7m" event={"ID":"0d350342-abc4-4bab-a5f5-56c03b02cac9","Type":"ContainerStarted","Data":"7638408e009cf37ede10bf5b561833cdc260dac19ecb3fab63baa7dd5db23c7f"} Jul 16 12:50:03.918558 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:03.918470 2565 generic.go:356] "Generic (PLEG): container finished" podID="a124a31a-c7e9-497a-b695-871976c48fa7" containerID="5dfe8d4c1f1de1a1ac8399401ff0a38cbf3714cfa278ec8ef31e7410d68106c2" exitCode=0 Jul 16 12:50:03.919014 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:03.918552 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" event={"ID":"a124a31a-c7e9-497a-b695-871976c48fa7","Type":"ContainerDied","Data":"5dfe8d4c1f1de1a1ac8399401ff0a38cbf3714cfa278ec8ef31e7410d68106c2"} Jul 16 12:50:03.920370 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:03.920347 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-2nf2r" event={"ID":"611dc307-70da-48f4-bb2a-f4697e55cf1c","Type":"ContainerStarted","Data":"e24e3a9c97c563b7dd5ea1b3c08ccc03c333ecafe81fba19ab6fb659e7cff803"} Jul 16 12:50:03.920503 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:03.920396 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-2nf2r" Jul 16 12:50:03.921909 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:03.921886 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nlj7m" event={"ID":"0d350342-abc4-4bab-a5f5-56c03b02cac9","Type":"ContainerStarted","Data":"cbe14c82fe923adffb501e065b2a18032cfbe224e4148c51f7372ed7f9ea1638"} Jul 16 12:50:03.971705 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:03.971658 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-2nf2r" podStartSLOduration=1.286588399 podStartE2EDuration="4.971641434s" totalImagesPullingTime="3.685053035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:50:03.970650685 +0000 UTC m=+311.965791445" watchObservedRunningTime="2026-07-16 12:50:03.971641434 +0000 UTC m=+311.966782193" Jul 16 12:50:03.998598 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:03.998548 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nlj7m" podStartSLOduration=0.987708797 podStartE2EDuration="2.998525124s" totalImagesPullingTime="2.010816327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:50:03.996440552 +0000 UTC m=+311.991581311" watchObservedRunningTime="2026-07-16 12:50:03.998525124 +0000 UTC m=+311.993665885" Jul 16 12:50:04.928517 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:04.928430 2565 generic.go:356] "Generic (PLEG): container finished" podID="a124a31a-c7e9-497a-b695-871976c48fa7" containerID="3a9503b76ad21afe8c63bc7e8b80ead2163a90252f50e65c704bb9563c30ccf7" exitCode=0 Jul 16 12:50:04.928880 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:04.928514 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" event={"ID":"a124a31a-c7e9-497a-b695-871976c48fa7","Type":"ContainerDied","Data":"3a9503b76ad21afe8c63bc7e8b80ead2163a90252f50e65c704bb9563c30ccf7"} Jul 16 12:50:06.061182 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:06.061160 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" Jul 16 12:50:06.214768 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:06.214695 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a124a31a-c7e9-497a-b695-871976c48fa7-bundle\") pod \"a124a31a-c7e9-497a-b695-871976c48fa7\" (UID: \"a124a31a-c7e9-497a-b695-871976c48fa7\") " Jul 16 12:50:06.214892 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:06.214809 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a124a31a-c7e9-497a-b695-871976c48fa7-util\") pod \"a124a31a-c7e9-497a-b695-871976c48fa7\" (UID: \"a124a31a-c7e9-497a-b695-871976c48fa7\") " Jul 16 12:50:06.214892 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:06.214857 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b22hx\" (UniqueName: \"kubernetes.io/projected/a124a31a-c7e9-497a-b695-871976c48fa7-kube-api-access-b22hx\") pod \"a124a31a-c7e9-497a-b695-871976c48fa7\" (UID: \"a124a31a-c7e9-497a-b695-871976c48fa7\") " Jul 16 12:50:06.215113 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:06.215092 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a124a31a-c7e9-497a-b695-871976c48fa7-bundle" (OuterVolumeSpecName: "bundle") pod "a124a31a-c7e9-497a-b695-871976c48fa7" (UID: "a124a31a-c7e9-497a-b695-871976c48fa7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:50:06.217068 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:06.217048 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a124a31a-c7e9-497a-b695-871976c48fa7-kube-api-access-b22hx" (OuterVolumeSpecName: "kube-api-access-b22hx") pod "a124a31a-c7e9-497a-b695-871976c48fa7" (UID: "a124a31a-c7e9-497a-b695-871976c48fa7"). InnerVolumeSpecName "kube-api-access-b22hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:50:06.219648 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:06.219626 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a124a31a-c7e9-497a-b695-871976c48fa7-util" (OuterVolumeSpecName: "util") pod "a124a31a-c7e9-497a-b695-871976c48fa7" (UID: "a124a31a-c7e9-497a-b695-871976c48fa7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:50:06.316139 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:06.316098 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a124a31a-c7e9-497a-b695-871976c48fa7-bundle\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:50:06.316139 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:06.316131 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a124a31a-c7e9-497a-b695-871976c48fa7-util\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:50:06.316139 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:06.316140 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b22hx\" (UniqueName: \"kubernetes.io/projected/a124a31a-c7e9-497a-b695-871976c48fa7-kube-api-access-b22hx\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:50:06.944001 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:06.943915 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" event={"ID":"a124a31a-c7e9-497a-b695-871976c48fa7","Type":"ContainerDied","Data":"fdba0562ee7f05336ccbf960a6e865cd5478e4dfcc6921e202579cdb4b6dbbc9"} Jul 16 12:50:06.944001 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:06.943942 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrgxs" Jul 16 12:50:06.944001 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:06.943974 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdba0562ee7f05336ccbf960a6e865cd5478e4dfcc6921e202579cdb4b6dbbc9" Jul 16 12:50:09.930964 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:09.930936 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-2nf2r" Jul 16 12:50:12.910615 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:12.910582 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln64s"] Jul 16 12:50:12.911059 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:12.910876 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a124a31a-c7e9-497a-b695-871976c48fa7" containerName="extract" Jul 16 12:50:12.911059 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:12.910886 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a124a31a-c7e9-497a-b695-871976c48fa7" containerName="extract" Jul 16 12:50:12.911059 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:12.910901 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a124a31a-c7e9-497a-b695-871976c48fa7" containerName="pull" Jul 16 12:50:12.911059 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:12.910906 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a124a31a-c7e9-497a-b695-871976c48fa7" containerName="pull" Jul 16 12:50:12.911059 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:12.910913 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a124a31a-c7e9-497a-b695-871976c48fa7" containerName="util" Jul 16 12:50:12.911059 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:12.910918 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a124a31a-c7e9-497a-b695-871976c48fa7" containerName="util" Jul 16 12:50:12.911059 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:12.910970 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a124a31a-c7e9-497a-b695-871976c48fa7" containerName="extract" Jul 16 12:50:12.914118 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:12.914103 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln64s" Jul 16 12:50:12.916520 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:12.916503 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 12:50:12.917376 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:12.917359 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:50:12.917484 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:12.917373 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-tsrlb\"" Jul 16 12:50:12.926597 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:12.926574 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln64s"] Jul 16 12:50:12.966933 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:12.966901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bd67b4b-b251-4f3b-92c2-a86695e53c8e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ln64s\" (UID: \"8bd67b4b-b251-4f3b-92c2-a86695e53c8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln64s" Jul 16 12:50:12.967036 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:12.966964 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgx27\" (UniqueName: \"kubernetes.io/projected/8bd67b4b-b251-4f3b-92c2-a86695e53c8e-kube-api-access-dgx27\") pod \"openshift-lws-operator-fd8ccff4c-ln64s\" (UID: \"8bd67b4b-b251-4f3b-92c2-a86695e53c8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln64s" Jul 16 12:50:13.067598 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:13.067566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgx27\" (UniqueName: \"kubernetes.io/projected/8bd67b4b-b251-4f3b-92c2-a86695e53c8e-kube-api-access-dgx27\") pod \"openshift-lws-operator-fd8ccff4c-ln64s\" (UID: \"8bd67b4b-b251-4f3b-92c2-a86695e53c8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln64s" Jul 16 12:50:13.067778 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:13.067653 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bd67b4b-b251-4f3b-92c2-a86695e53c8e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ln64s\" (UID: \"8bd67b4b-b251-4f3b-92c2-a86695e53c8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln64s" Jul 16 12:50:13.068021 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:13.068002 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bd67b4b-b251-4f3b-92c2-a86695e53c8e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ln64s\" (UID: \"8bd67b4b-b251-4f3b-92c2-a86695e53c8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln64s" Jul 16 12:50:13.077126 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:13.077101 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgx27\" (UniqueName: \"kubernetes.io/projected/8bd67b4b-b251-4f3b-92c2-a86695e53c8e-kube-api-access-dgx27\") pod \"openshift-lws-operator-fd8ccff4c-ln64s\" (UID: \"8bd67b4b-b251-4f3b-92c2-a86695e53c8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln64s" Jul 16 12:50:13.223686 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:13.223607 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln64s" Jul 16 12:50:13.362547 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:13.362519 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln64s"] Jul 16 12:50:13.364917 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:50:13.364887 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bd67b4b_b251_4f3b_92c2_a86695e53c8e.slice/crio-555b860b15e02fe6bcdf701faccd2344d5e314edb547000bbc3df7a44dbe54ad WatchSource:0}: Error finding container 555b860b15e02fe6bcdf701faccd2344d5e314edb547000bbc3df7a44dbe54ad: Status 404 returned error can't find the container with id 555b860b15e02fe6bcdf701faccd2344d5e314edb547000bbc3df7a44dbe54ad Jul 16 12:50:13.974189 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:13.974158 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln64s" event={"ID":"8bd67b4b-b251-4f3b-92c2-a86695e53c8e","Type":"ContainerStarted","Data":"555b860b15e02fe6bcdf701faccd2344d5e314edb547000bbc3df7a44dbe54ad"} Jul 16 12:50:15.986340 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:15.986302 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln64s" event={"ID":"8bd67b4b-b251-4f3b-92c2-a86695e53c8e","Type":"ContainerStarted","Data":"3adc2d3365fc00dcdfc6642d2d6864416fedb8bfc7d0a095ac376abd44342ee8"} Jul 16 12:50:16.007707 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:16.007662 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ln64s" podStartSLOduration=1.532546592 podStartE2EDuration="4.007646717s" totalImagesPullingTime="2.475100125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:50:16.006062516 +0000 UTC m=+324.001203291" watchObservedRunningTime="2026-07-16 12:50:16.007646717 +0000 UTC m=+324.002787476" Jul 16 12:50:18.506636 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.506604 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr"] Jul 16 12:50:18.510150 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.510132 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" Jul 16 12:50:18.512895 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.512862 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 12:50:18.513012 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.512907 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 12:50:18.513775 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.513754 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5v8g6\"" Jul 16 12:50:18.522521 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.522498 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr"] Jul 16 12:50:18.610475 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.610447 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9440531a-3078-40b4-b006-633972b831de-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr\" (UID: \"9440531a-3078-40b4-b006-633972b831de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" Jul 16 12:50:18.610636 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.610480 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9440531a-3078-40b4-b006-633972b831de-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr\" (UID: \"9440531a-3078-40b4-b006-633972b831de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" Jul 16 12:50:18.610636 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.610594 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn2m4\" (UniqueName: \"kubernetes.io/projected/9440531a-3078-40b4-b006-633972b831de-kube-api-access-kn2m4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr\" (UID: \"9440531a-3078-40b4-b006-633972b831de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" Jul 16 12:50:18.710972 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.710934 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn2m4\" (UniqueName: \"kubernetes.io/projected/9440531a-3078-40b4-b006-633972b831de-kube-api-access-kn2m4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr\" (UID: \"9440531a-3078-40b4-b006-633972b831de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" Jul 16 12:50:18.710972 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.710975 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9440531a-3078-40b4-b006-633972b831de-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr\" (UID: \"9440531a-3078-40b4-b006-633972b831de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" Jul 16 12:50:18.711141 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.710994 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9440531a-3078-40b4-b006-633972b831de-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr\" (UID: \"9440531a-3078-40b4-b006-633972b831de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" Jul 16 12:50:18.711417 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.711401 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9440531a-3078-40b4-b006-633972b831de-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr\" (UID: \"9440531a-3078-40b4-b006-633972b831de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" Jul 16 12:50:18.711471 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.711420 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9440531a-3078-40b4-b006-633972b831de-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr\" (UID: \"9440531a-3078-40b4-b006-633972b831de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" Jul 16 12:50:18.720550 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.720529 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn2m4\" (UniqueName: \"kubernetes.io/projected/9440531a-3078-40b4-b006-633972b831de-kube-api-access-kn2m4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr\" (UID: \"9440531a-3078-40b4-b006-633972b831de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" Jul 16 12:50:18.820147 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.820126 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" Jul 16 12:50:18.958736 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.958678 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr"] Jul 16 12:50:18.962160 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:50:18.962125 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9440531a_3078_40b4_b006_633972b831de.slice/crio-ffa4589c68c0f8222149bb1b28533909120a3136d91e9de99475d8ce5af92491 WatchSource:0}: Error finding container ffa4589c68c0f8222149bb1b28533909120a3136d91e9de99475d8ce5af92491: Status 404 returned error can't find the container with id ffa4589c68c0f8222149bb1b28533909120a3136d91e9de99475d8ce5af92491 Jul 16 12:50:19.000021 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:18.999995 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" event={"ID":"9440531a-3078-40b4-b006-633972b831de","Type":"ContainerStarted","Data":"ffa4589c68c0f8222149bb1b28533909120a3136d91e9de99475d8ce5af92491"} Jul 16 12:50:20.005753 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:20.005668 2565 generic.go:356] "Generic (PLEG): container finished" podID="9440531a-3078-40b4-b006-633972b831de" containerID="20b8bfe24071210d25be21d41911d9d3e4ad8884d46e34c75d9143bd88b4a039" exitCode=0 Jul 16 12:50:20.006142 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:20.005751 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" event={"ID":"9440531a-3078-40b4-b006-633972b831de","Type":"ContainerDied","Data":"20b8bfe24071210d25be21d41911d9d3e4ad8884d46e34c75d9143bd88b4a039"} Jul 16 12:50:21.011916 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:21.011886 2565 generic.go:356] "Generic (PLEG): container finished" podID="9440531a-3078-40b4-b006-633972b831de" containerID="535a25f3baec688a58899eaf256a2b03fd0fe76b347fc9411161f045b2afde1b" exitCode=0 Jul 16 12:50:21.012318 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:21.011965 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" event={"ID":"9440531a-3078-40b4-b006-633972b831de","Type":"ContainerDied","Data":"535a25f3baec688a58899eaf256a2b03fd0fe76b347fc9411161f045b2afde1b"} Jul 16 12:50:22.017904 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:22.017873 2565 generic.go:356] "Generic (PLEG): container finished" podID="9440531a-3078-40b4-b006-633972b831de" containerID="ac8b17a11fe6d8793469c6b72f588c872ffd1b676217041995a06a1070d8cf9a" exitCode=0 Jul 16 12:50:22.018345 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:22.017963 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" event={"ID":"9440531a-3078-40b4-b006-633972b831de","Type":"ContainerDied","Data":"ac8b17a11fe6d8793469c6b72f588c872ffd1b676217041995a06a1070d8cf9a"} Jul 16 12:50:23.153696 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:23.153672 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" Jul 16 12:50:23.248771 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:23.248732 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9440531a-3078-40b4-b006-633972b831de-util\") pod \"9440531a-3078-40b4-b006-633972b831de\" (UID: \"9440531a-3078-40b4-b006-633972b831de\") " Jul 16 12:50:23.248933 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:23.248780 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9440531a-3078-40b4-b006-633972b831de-bundle\") pod \"9440531a-3078-40b4-b006-633972b831de\" (UID: \"9440531a-3078-40b4-b006-633972b831de\") " Jul 16 12:50:23.248933 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:23.248859 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn2m4\" (UniqueName: \"kubernetes.io/projected/9440531a-3078-40b4-b006-633972b831de-kube-api-access-kn2m4\") pod \"9440531a-3078-40b4-b006-633972b831de\" (UID: \"9440531a-3078-40b4-b006-633972b831de\") " Jul 16 12:50:23.249460 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:23.249428 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9440531a-3078-40b4-b006-633972b831de-bundle" (OuterVolumeSpecName: "bundle") pod "9440531a-3078-40b4-b006-633972b831de" (UID: "9440531a-3078-40b4-b006-633972b831de"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:50:23.251053 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:23.251031 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9440531a-3078-40b4-b006-633972b831de-kube-api-access-kn2m4" (OuterVolumeSpecName: "kube-api-access-kn2m4") pod "9440531a-3078-40b4-b006-633972b831de" (UID: "9440531a-3078-40b4-b006-633972b831de"). InnerVolumeSpecName "kube-api-access-kn2m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:50:23.254588 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:23.254562 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9440531a-3078-40b4-b006-633972b831de-util" (OuterVolumeSpecName: "util") pod "9440531a-3078-40b4-b006-633972b831de" (UID: "9440531a-3078-40b4-b006-633972b831de"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:50:23.349455 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:23.349392 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn2m4\" (UniqueName: \"kubernetes.io/projected/9440531a-3078-40b4-b006-633972b831de-kube-api-access-kn2m4\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:50:23.349455 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:23.349418 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9440531a-3078-40b4-b006-633972b831de-util\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:50:23.349455 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:23.349431 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9440531a-3078-40b4-b006-633972b831de-bundle\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:50:24.028541 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:24.028509 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" event={"ID":"9440531a-3078-40b4-b006-633972b831de","Type":"ContainerDied","Data":"ffa4589c68c0f8222149bb1b28533909120a3136d91e9de99475d8ce5af92491"} Jul 16 12:50:24.028541 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:24.028535 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7f8kr" Jul 16 12:50:24.028755 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:24.028554 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffa4589c68c0f8222149bb1b28533909120a3136d91e9de99475d8ce5af92491" Jul 16 12:50:35.439708 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.439673 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8"] Jul 16 12:50:35.440076 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.439959 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9440531a-3078-40b4-b006-633972b831de" containerName="util" Jul 16 12:50:35.440076 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.439969 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9440531a-3078-40b4-b006-633972b831de" containerName="util" Jul 16 12:50:35.440076 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.439977 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9440531a-3078-40b4-b006-633972b831de" containerName="extract" Jul 16 12:50:35.440076 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.439982 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9440531a-3078-40b4-b006-633972b831de" containerName="extract" Jul 16 12:50:35.440076 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.439994 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9440531a-3078-40b4-b006-633972b831de" containerName="pull" Jul 16 12:50:35.440076 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.439999 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9440531a-3078-40b4-b006-633972b831de" containerName="pull" Jul 16 12:50:35.440076 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.440059 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9440531a-3078-40b4-b006-633972b831de" containerName="extract" Jul 16 12:50:35.445497 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.445476 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.448320 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.448298 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 12:50:35.448455 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.448358 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 12:50:35.448455 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.448382 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-v8qn5\"" Jul 16 12:50:35.448583 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.448461 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 12:50:35.448659 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.448627 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 12:50:35.462706 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.462685 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8"] Jul 16 12:50:35.559691 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.559654 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3277b759-4517-4cb2-a78b-108ca160674d-webhook-cert\") pod \"opendatahub-operator-controller-manager-9d668944c-n5ft8\" (UID: \"3277b759-4517-4cb2-a78b-108ca160674d\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.559867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.559701 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3277b759-4517-4cb2-a78b-108ca160674d-tmp\") pod \"opendatahub-operator-controller-manager-9d668944c-n5ft8\" (UID: \"3277b759-4517-4cb2-a78b-108ca160674d\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.559867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.559721 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3277b759-4517-4cb2-a78b-108ca160674d-manifests\") pod \"opendatahub-operator-controller-manager-9d668944c-n5ft8\" (UID: \"3277b759-4517-4cb2-a78b-108ca160674d\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.559867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.559832 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l9z8\" (UniqueName: \"kubernetes.io/projected/3277b759-4517-4cb2-a78b-108ca160674d-kube-api-access-2l9z8\") pod \"opendatahub-operator-controller-manager-9d668944c-n5ft8\" (UID: \"3277b759-4517-4cb2-a78b-108ca160674d\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.559971 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.559872 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3277b759-4517-4cb2-a78b-108ca160674d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9d668944c-n5ft8\" (UID: \"3277b759-4517-4cb2-a78b-108ca160674d\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.661251 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.661199 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3277b759-4517-4cb2-a78b-108ca160674d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9d668944c-n5ft8\" (UID: \"3277b759-4517-4cb2-a78b-108ca160674d\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.661251 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.661251 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3277b759-4517-4cb2-a78b-108ca160674d-webhook-cert\") pod \"opendatahub-operator-controller-manager-9d668944c-n5ft8\" (UID: \"3277b759-4517-4cb2-a78b-108ca160674d\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.661463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.661283 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3277b759-4517-4cb2-a78b-108ca160674d-tmp\") pod \"opendatahub-operator-controller-manager-9d668944c-n5ft8\" (UID: \"3277b759-4517-4cb2-a78b-108ca160674d\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.661463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.661312 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3277b759-4517-4cb2-a78b-108ca160674d-manifests\") pod \"opendatahub-operator-controller-manager-9d668944c-n5ft8\" (UID: \"3277b759-4517-4cb2-a78b-108ca160674d\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.661463 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.661358 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9z8\" (UniqueName: \"kubernetes.io/projected/3277b759-4517-4cb2-a78b-108ca160674d-kube-api-access-2l9z8\") pod \"opendatahub-operator-controller-manager-9d668944c-n5ft8\" (UID: \"3277b759-4517-4cb2-a78b-108ca160674d\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.661772 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.661736 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3277b759-4517-4cb2-a78b-108ca160674d-manifests\") pod \"opendatahub-operator-controller-manager-9d668944c-n5ft8\" (UID: \"3277b759-4517-4cb2-a78b-108ca160674d\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.663710 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.663690 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3277b759-4517-4cb2-a78b-108ca160674d-tmp\") pod \"opendatahub-operator-controller-manager-9d668944c-n5ft8\" (UID: \"3277b759-4517-4cb2-a78b-108ca160674d\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.663972 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.663952 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3277b759-4517-4cb2-a78b-108ca160674d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9d668944c-n5ft8\" (UID: \"3277b759-4517-4cb2-a78b-108ca160674d\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.663972 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.663966 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3277b759-4517-4cb2-a78b-108ca160674d-webhook-cert\") pod \"opendatahub-operator-controller-manager-9d668944c-n5ft8\" (UID: \"3277b759-4517-4cb2-a78b-108ca160674d\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.679065 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.679045 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l9z8\" (UniqueName: \"kubernetes.io/projected/3277b759-4517-4cb2-a78b-108ca160674d-kube-api-access-2l9z8\") pod \"opendatahub-operator-controller-manager-9d668944c-n5ft8\" (UID: \"3277b759-4517-4cb2-a78b-108ca160674d\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.755532 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.755449 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:35.901831 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:35.901801 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8"] Jul 16 12:50:35.904885 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:50:35.904856 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3277b759_4517_4cb2_a78b_108ca160674d.slice/crio-02728d473bcc425ca6a9501f912f2f7e454c6597572f4178c85719416165c8ca WatchSource:0}: Error finding container 02728d473bcc425ca6a9501f912f2f7e454c6597572f4178c85719416165c8ca: Status 404 returned error can't find the container with id 02728d473bcc425ca6a9501f912f2f7e454c6597572f4178c85719416165c8ca Jul 16 12:50:36.086108 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:36.086072 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" event={"ID":"3277b759-4517-4cb2-a78b-108ca160674d","Type":"ContainerStarted","Data":"02728d473bcc425ca6a9501f912f2f7e454c6597572f4178c85719416165c8ca"} Jul 16 12:50:40.111714 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:40.111672 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" event={"ID":"3277b759-4517-4cb2-a78b-108ca160674d","Type":"ContainerStarted","Data":"ea746eceb4e82e8e39fdb63dfd006d8194bbd11460810d29b357e4477b7b43cf"} Jul 16 12:50:41.120322 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:41.120289 2565 generic.go:356] "Generic (PLEG): container finished" podID="3277b759-4517-4cb2-a78b-108ca160674d" containerID="ea746eceb4e82e8e39fdb63dfd006d8194bbd11460810d29b357e4477b7b43cf" exitCode=0 Jul 16 12:50:41.120765 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:41.120376 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" event={"ID":"3277b759-4517-4cb2-a78b-108ca160674d","Type":"ContainerDied","Data":"ea746eceb4e82e8e39fdb63dfd006d8194bbd11460810d29b357e4477b7b43cf"} Jul 16 12:50:42.126642 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:42.126547 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" event={"ID":"3277b759-4517-4cb2-a78b-108ca160674d","Type":"ContainerStarted","Data":"812f809a114a7a6d4933d8862610d48da9d1abf01c2722c712e43cde440971ad"} Jul 16 12:50:42.126642 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:42.126623 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:42.163592 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:42.163543 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" podStartSLOduration=2.039439743 podStartE2EDuration="7.163526873s" totalImagesPullingTime="4.042883841s" totalInitContainerRuntime="1.081203289s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:50:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:50:42.161786019 +0000 UTC m=+350.156926780" watchObservedRunningTime="2026-07-16 12:50:42.163526873 +0000 UTC m=+350.158667633" Jul 16 12:50:53.133755 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.133666 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-n5ft8" Jul 16 12:50:53.454198 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.454113 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz"] Jul 16 12:50:53.459452 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.459429 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" Jul 16 12:50:53.461894 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.461857 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 16 12:50:53.462009 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.461936 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qwbmj\"" Jul 16 12:50:53.462009 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.461962 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 16 12:50:53.462143 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.461963 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 16 12:50:53.469487 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.469466 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz"] Jul 16 12:50:53.612321 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.612283 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bc2c5cdd-3a95-4ce6-afb3-0f3240733a75-manager-config\") pod \"lws-controller-manager-5864f4f95b-fpwjz\" (UID: \"bc2c5cdd-3a95-4ce6-afb3-0f3240733a75\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" Jul 16 12:50:53.612321 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.612326 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc2c5cdd-3a95-4ce6-afb3-0f3240733a75-metrics-cert\") pod \"lws-controller-manager-5864f4f95b-fpwjz\" (UID: \"bc2c5cdd-3a95-4ce6-afb3-0f3240733a75\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" Jul 16 12:50:53.612586 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.612350 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjwq8\" (UniqueName: \"kubernetes.io/projected/bc2c5cdd-3a95-4ce6-afb3-0f3240733a75-kube-api-access-gjwq8\") pod \"lws-controller-manager-5864f4f95b-fpwjz\" (UID: \"bc2c5cdd-3a95-4ce6-afb3-0f3240733a75\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" Jul 16 12:50:53.612586 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.612403 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc2c5cdd-3a95-4ce6-afb3-0f3240733a75-cert\") pod \"lws-controller-manager-5864f4f95b-fpwjz\" (UID: \"bc2c5cdd-3a95-4ce6-afb3-0f3240733a75\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" Jul 16 12:50:53.713548 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.713466 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bc2c5cdd-3a95-4ce6-afb3-0f3240733a75-manager-config\") pod \"lws-controller-manager-5864f4f95b-fpwjz\" (UID: \"bc2c5cdd-3a95-4ce6-afb3-0f3240733a75\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" Jul 16 12:50:53.713548 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.713500 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc2c5cdd-3a95-4ce6-afb3-0f3240733a75-metrics-cert\") pod \"lws-controller-manager-5864f4f95b-fpwjz\" (UID: \"bc2c5cdd-3a95-4ce6-afb3-0f3240733a75\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" Jul 16 12:50:53.713548 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.713516 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjwq8\" (UniqueName: \"kubernetes.io/projected/bc2c5cdd-3a95-4ce6-afb3-0f3240733a75-kube-api-access-gjwq8\") pod \"lws-controller-manager-5864f4f95b-fpwjz\" (UID: \"bc2c5cdd-3a95-4ce6-afb3-0f3240733a75\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" Jul 16 12:50:53.713548 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.713534 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc2c5cdd-3a95-4ce6-afb3-0f3240733a75-cert\") pod \"lws-controller-manager-5864f4f95b-fpwjz\" (UID: \"bc2c5cdd-3a95-4ce6-afb3-0f3240733a75\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" Jul 16 12:50:53.714287 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.714263 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bc2c5cdd-3a95-4ce6-afb3-0f3240733a75-manager-config\") pod \"lws-controller-manager-5864f4f95b-fpwjz\" (UID: \"bc2c5cdd-3a95-4ce6-afb3-0f3240733a75\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" Jul 16 12:50:53.716047 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.716023 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc2c5cdd-3a95-4ce6-afb3-0f3240733a75-metrics-cert\") pod \"lws-controller-manager-5864f4f95b-fpwjz\" (UID: \"bc2c5cdd-3a95-4ce6-afb3-0f3240733a75\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" Jul 16 12:50:53.716143 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.716068 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc2c5cdd-3a95-4ce6-afb3-0f3240733a75-cert\") pod \"lws-controller-manager-5864f4f95b-fpwjz\" (UID: \"bc2c5cdd-3a95-4ce6-afb3-0f3240733a75\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" Jul 16 12:50:53.724479 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.724452 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjwq8\" (UniqueName: \"kubernetes.io/projected/bc2c5cdd-3a95-4ce6-afb3-0f3240733a75-kube-api-access-gjwq8\") pod \"lws-controller-manager-5864f4f95b-fpwjz\" (UID: \"bc2c5cdd-3a95-4ce6-afb3-0f3240733a75\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" Jul 16 12:50:53.768635 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.768602 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" Jul 16 12:50:53.907811 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:53.907781 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz"] Jul 16 12:50:53.910993 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:50:53.910972 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2c5cdd_3a95_4ce6_afb3_0f3240733a75.slice/crio-a680774bf4baa3d77d7eef5bffd5f43b5ab885de4376ec967380c6bec7b8eb7c WatchSource:0}: Error finding container a680774bf4baa3d77d7eef5bffd5f43b5ab885de4376ec967380c6bec7b8eb7c: Status 404 returned error can't find the container with id a680774bf4baa3d77d7eef5bffd5f43b5ab885de4376ec967380c6bec7b8eb7c Jul 16 12:50:54.184782 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:54.184751 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" event={"ID":"bc2c5cdd-3a95-4ce6-afb3-0f3240733a75","Type":"ContainerStarted","Data":"a680774bf4baa3d77d7eef5bffd5f43b5ab885de4376ec967380c6bec7b8eb7c"} Jul 16 12:50:55.191866 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:55.191829 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" event={"ID":"bc2c5cdd-3a95-4ce6-afb3-0f3240733a75","Type":"ContainerStarted","Data":"455813448b9113a13b62b671281da352c8fc01561490dcf62d7b015b7d4d453e"} Jul 16 12:50:55.192269 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:55.191995 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" Jul 16 12:50:55.271263 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:50:55.271189 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" podStartSLOduration=1.107253107 podStartE2EDuration="2.271166628s" totalImagesPullingTime="1.163913521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:50:55.269224154 +0000 UTC m=+363.264364939" watchObservedRunningTime="2026-07-16 12:50:55.271166628 +0000 UTC m=+363.266307389" Jul 16 12:51:06.198220 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:06.198187 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-fpwjz" Jul 16 12:51:36.075616 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.075584 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm"] Jul 16 12:51:36.077768 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.077749 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.081540 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.081516 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 12:51:36.081643 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.081624 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4rzns\"" Jul 16 12:51:36.095949 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.095930 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm"] Jul 16 12:51:36.185652 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.185620 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.185652 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.185656 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.185893 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.185686 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.185893 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.185772 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.185893 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.185806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtktk\" (UniqueName: \"kubernetes.io/projected/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-kube-api-access-qtktk\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.185893 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.185829 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.185893 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.185849 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.186078 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.185942 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.186078 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.185972 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.287349 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.287309 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.287577 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.287366 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.287577 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.287388 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtktk\" (UniqueName: \"kubernetes.io/projected/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-kube-api-access-qtktk\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.287577 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.287409 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.287577 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.287428 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.287577 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.287462 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.287577 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.287478 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.287881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.287696 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.287881 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.287738 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.287985 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.287947 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.288031 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.288012 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.288081 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.288059 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.288266 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.288222 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.288443 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.288418 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.289852 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.289829 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.290143 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.290125 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.300534 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.300498 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.300671 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.300656 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtktk\" (UniqueName: \"kubernetes.io/projected/1dc9d367-ab3a-4432-b7f1-47a17d8f0c89-kube-api-access-qtktk\") pod \"data-science-gateway-data-science-gateway-class-6576859cf44r4nm\" (UID: \"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.388423 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.388328 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:36.536018 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:36.535990 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm"] Jul 16 12:51:36.539440 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:51:36.539413 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc9d367_ab3a_4432_b7f1_47a17d8f0c89.slice/crio-52b2a7f8433b22b71a46b775a3d0f9008f337a6bf9b8f0b941fc0295321c0ccd WatchSource:0}: Error finding container 52b2a7f8433b22b71a46b775a3d0f9008f337a6bf9b8f0b941fc0295321c0ccd: Status 404 returned error can't find the container with id 52b2a7f8433b22b71a46b775a3d0f9008f337a6bf9b8f0b941fc0295321c0ccd Jul 16 12:51:37.393042 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:37.393004 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" event={"ID":"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89","Type":"ContainerStarted","Data":"52b2a7f8433b22b71a46b775a3d0f9008f337a6bf9b8f0b941fc0295321c0ccd"} Jul 16 12:51:39.272834 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:39.272793 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 16 12:51:39.273070 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:39.272870 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 16 12:51:39.273070 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:39.272900 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 16 12:51:39.404215 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:39.404179 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" event={"ID":"1dc9d367-ab3a-4432-b7f1-47a17d8f0c89","Type":"ContainerStarted","Data":"8d214e6692dcc633e76fb8df2ea0989c68fb52af44199cbb40e8e2b2f7ac4147"} Jul 16 12:51:39.432180 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:39.432134 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" podStartSLOduration=0.700941398 podStartE2EDuration="3.432114192s" totalImagesPullingTime="2.731172794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:51:39.42906962 +0000 UTC m=+407.424210374" watchObservedRunningTime="2026-07-16 12:51:39.432114192 +0000 UTC m=+407.427254955" Jul 16 12:51:40.388687 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:40.388648 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:40.393784 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:40.393759 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:40.409092 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:40.409065 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:40.409977 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:40.409956 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf44r4nm" Jul 16 12:51:56.895719 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:56.895685 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d8c8v"] Jul 16 12:51:56.899161 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:56.899145 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d8c8v" Jul 16 12:51:56.901721 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:56.901688 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 12:51:56.901721 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:56.901701 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 12:51:56.902517 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:56.902500 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-dpfz9\"" Jul 16 12:51:56.912967 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:56.912941 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d8c8v"] Jul 16 12:51:56.977568 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:56.977530 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mlmz\" (UniqueName: \"kubernetes.io/projected/18894e52-9d11-4a90-9a39-80d9466aadf1-kube-api-access-7mlmz\") pod \"kuadrant-operator-catalog-d8c8v\" (UID: \"18894e52-9d11-4a90-9a39-80d9466aadf1\") " pod="kuadrant-system/kuadrant-operator-catalog-d8c8v" Jul 16 12:51:57.078348 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:57.078315 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mlmz\" (UniqueName: \"kubernetes.io/projected/18894e52-9d11-4a90-9a39-80d9466aadf1-kube-api-access-7mlmz\") pod \"kuadrant-operator-catalog-d8c8v\" (UID: \"18894e52-9d11-4a90-9a39-80d9466aadf1\") " pod="kuadrant-system/kuadrant-operator-catalog-d8c8v" Jul 16 12:51:57.116158 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:57.116121 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mlmz\" (UniqueName: \"kubernetes.io/projected/18894e52-9d11-4a90-9a39-80d9466aadf1-kube-api-access-7mlmz\") pod \"kuadrant-operator-catalog-d8c8v\" (UID: \"18894e52-9d11-4a90-9a39-80d9466aadf1\") " pod="kuadrant-system/kuadrant-operator-catalog-d8c8v" Jul 16 12:51:57.190047 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:57.189976 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d8c8v"] Jul 16 12:51:57.190277 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:57.190264 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d8c8v" Jul 16 12:51:57.329013 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:57.328989 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d8c8v"] Jul 16 12:51:57.330102 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:51:57.330077 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18894e52_9d11_4a90_9a39_80d9466aadf1.slice/crio-930cbb9e09a01be7e8f55cc6cd4ea534be2eeab5518d603842b5ca865c3718ad WatchSource:0}: Error finding container 930cbb9e09a01be7e8f55cc6cd4ea534be2eeab5518d603842b5ca865c3718ad: Status 404 returned error can't find the container with id 930cbb9e09a01be7e8f55cc6cd4ea534be2eeab5518d603842b5ca865c3718ad Jul 16 12:51:57.407599 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:57.407566 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8c7pg"] Jul 16 12:51:57.410089 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:57.410072 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8c7pg" Jul 16 12:51:57.424367 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:57.424343 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8c7pg"] Jul 16 12:51:57.482414 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:57.482331 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szvww\" (UniqueName: \"kubernetes.io/projected/a40b7195-74d8-4613-8409-53054056fdb2-kube-api-access-szvww\") pod \"kuadrant-operator-catalog-8c7pg\" (UID: \"a40b7195-74d8-4613-8409-53054056fdb2\") " pod="kuadrant-system/kuadrant-operator-catalog-8c7pg" Jul 16 12:51:57.492710 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:57.492682 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d8c8v" event={"ID":"18894e52-9d11-4a90-9a39-80d9466aadf1","Type":"ContainerStarted","Data":"930cbb9e09a01be7e8f55cc6cd4ea534be2eeab5518d603842b5ca865c3718ad"} Jul 16 12:51:57.583261 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:57.583196 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szvww\" (UniqueName: \"kubernetes.io/projected/a40b7195-74d8-4613-8409-53054056fdb2-kube-api-access-szvww\") pod \"kuadrant-operator-catalog-8c7pg\" (UID: \"a40b7195-74d8-4613-8409-53054056fdb2\") " pod="kuadrant-system/kuadrant-operator-catalog-8c7pg" Jul 16 12:51:57.593158 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:57.593128 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szvww\" (UniqueName: \"kubernetes.io/projected/a40b7195-74d8-4613-8409-53054056fdb2-kube-api-access-szvww\") pod \"kuadrant-operator-catalog-8c7pg\" (UID: \"a40b7195-74d8-4613-8409-53054056fdb2\") " pod="kuadrant-system/kuadrant-operator-catalog-8c7pg" Jul 16 12:51:57.719523 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:57.719493 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8c7pg" Jul 16 12:51:57.883706 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:57.883665 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8c7pg"] Jul 16 12:51:57.886855 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:51:57.886798 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40b7195_74d8_4613_8409_53054056fdb2.slice/crio-8c5f6067638bfe074b9dcc7887d94a6d7b943370fe3177527c0eae70a7b411ff WatchSource:0}: Error finding container 8c5f6067638bfe074b9dcc7887d94a6d7b943370fe3177527c0eae70a7b411ff: Status 404 returned error can't find the container with id 8c5f6067638bfe074b9dcc7887d94a6d7b943370fe3177527c0eae70a7b411ff Jul 16 12:51:58.500580 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:58.500544 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8c7pg" event={"ID":"a40b7195-74d8-4613-8409-53054056fdb2","Type":"ContainerStarted","Data":"8c5f6067638bfe074b9dcc7887d94a6d7b943370fe3177527c0eae70a7b411ff"} Jul 16 12:51:59.507539 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:59.507507 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d8c8v" event={"ID":"18894e52-9d11-4a90-9a39-80d9466aadf1","Type":"ContainerStarted","Data":"84d70910f62e6db2ce4dc79bd978a20d6ffa85da74f93643c099b0d47872ac8c"} Jul 16 12:51:59.507975 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:59.507583 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-d8c8v" podUID="18894e52-9d11-4a90-9a39-80d9466aadf1" containerName="registry-server" containerID="cri-o://84d70910f62e6db2ce4dc79bd978a20d6ffa85da74f93643c099b0d47872ac8c" gracePeriod=2 Jul 16 12:51:59.509432 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:59.509403 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8c7pg" event={"ID":"a40b7195-74d8-4613-8409-53054056fdb2","Type":"ContainerStarted","Data":"b87b23a2d43463b57cad98df1e83f708e7a66748ab09cf755db8793d338d9f17"} Jul 16 12:51:59.530222 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:59.530175 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-d8c8v" podStartSLOduration=1.458972434 podStartE2EDuration="3.530155673s" totalImagesPullingTime="2.071183239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:51:59.529032877 +0000 UTC m=+427.524173638" watchObservedRunningTime="2026-07-16 12:51:59.530155673 +0000 UTC m=+427.525296433" Jul 16 12:51:59.551706 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:59.551657 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8c7pg" podStartSLOduration=1.036413228 podStartE2EDuration="2.551640972s" totalImagesPullingTime="1.515227744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:51:59.549071361 +0000 UTC m=+427.544212123" watchObservedRunningTime="2026-07-16 12:51:59.551640972 +0000 UTC m=+427.546781731" Jul 16 12:51:59.756434 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:59.752222 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-d8c8v_18894e52-9d11-4a90-9a39-80d9466aadf1/registry-server/0.log" Jul 16 12:51:59.757757 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:59.757726 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d8c8v" Jul 16 12:51:59.804146 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:59.804115 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mlmz\" (UniqueName: \"kubernetes.io/projected/18894e52-9d11-4a90-9a39-80d9466aadf1-kube-api-access-7mlmz\") pod \"18894e52-9d11-4a90-9a39-80d9466aadf1\" (UID: \"18894e52-9d11-4a90-9a39-80d9466aadf1\") " Jul 16 12:51:59.806424 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:59.806402 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18894e52-9d11-4a90-9a39-80d9466aadf1-kube-api-access-7mlmz" (OuterVolumeSpecName: "kube-api-access-7mlmz") pod "18894e52-9d11-4a90-9a39-80d9466aadf1" (UID: "18894e52-9d11-4a90-9a39-80d9466aadf1"). InnerVolumeSpecName "kube-api-access-7mlmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:51:59.905274 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:51:59.905222 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mlmz\" (UniqueName: \"kubernetes.io/projected/18894e52-9d11-4a90-9a39-80d9466aadf1-kube-api-access-7mlmz\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:00.515740 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:00.515712 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-d8c8v_18894e52-9d11-4a90-9a39-80d9466aadf1/registry-server/0.log" Jul 16 12:52:00.516264 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:00.515753 2565 generic.go:356] "Generic (PLEG): container finished" podID="18894e52-9d11-4a90-9a39-80d9466aadf1" containerID="84d70910f62e6db2ce4dc79bd978a20d6ffa85da74f93643c099b0d47872ac8c" exitCode=1 Jul 16 12:52:00.516264 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:00.515816 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d8c8v" Jul 16 12:52:00.516264 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:00.515845 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d8c8v" event={"ID":"18894e52-9d11-4a90-9a39-80d9466aadf1","Type":"ContainerDied","Data":"84d70910f62e6db2ce4dc79bd978a20d6ffa85da74f93643c099b0d47872ac8c"} Jul 16 12:52:00.516264 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:00.515922 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d8c8v" event={"ID":"18894e52-9d11-4a90-9a39-80d9466aadf1","Type":"ContainerDied","Data":"930cbb9e09a01be7e8f55cc6cd4ea534be2eeab5518d603842b5ca865c3718ad"} Jul 16 12:52:00.516264 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:00.515946 2565 scope.go:117] "RemoveContainer" containerID="84d70910f62e6db2ce4dc79bd978a20d6ffa85da74f93643c099b0d47872ac8c" Jul 16 12:52:00.525548 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:00.525531 2565 scope.go:117] "RemoveContainer" containerID="84d70910f62e6db2ce4dc79bd978a20d6ffa85da74f93643c099b0d47872ac8c" Jul 16 12:52:00.525790 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:52:00.525770 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d70910f62e6db2ce4dc79bd978a20d6ffa85da74f93643c099b0d47872ac8c\": container with ID starting with 84d70910f62e6db2ce4dc79bd978a20d6ffa85da74f93643c099b0d47872ac8c not found: ID does not exist" containerID="84d70910f62e6db2ce4dc79bd978a20d6ffa85da74f93643c099b0d47872ac8c" Jul 16 12:52:00.525846 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:00.525799 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d70910f62e6db2ce4dc79bd978a20d6ffa85da74f93643c099b0d47872ac8c"} err="failed to get container status \"84d70910f62e6db2ce4dc79bd978a20d6ffa85da74f93643c099b0d47872ac8c\": rpc error: code = NotFound desc = could not find container \"84d70910f62e6db2ce4dc79bd978a20d6ffa85da74f93643c099b0d47872ac8c\": container with ID starting with 84d70910f62e6db2ce4dc79bd978a20d6ffa85da74f93643c099b0d47872ac8c not found: ID does not exist" Jul 16 12:52:00.552096 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:00.552063 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d8c8v"] Jul 16 12:52:00.557660 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:00.557634 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d8c8v"] Jul 16 12:52:02.545172 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:02.545138 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18894e52-9d11-4a90-9a39-80d9466aadf1" path="/var/lib/kubelet/pods/18894e52-9d11-4a90-9a39-80d9466aadf1/volumes" Jul 16 12:52:03.372249 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.372201 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5549db9dd8-z4wts"] Jul 16 12:52:03.372702 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.372688 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18894e52-9d11-4a90-9a39-80d9466aadf1" containerName="registry-server" Jul 16 12:52:03.372748 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.372706 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="18894e52-9d11-4a90-9a39-80d9466aadf1" containerName="registry-server" Jul 16 12:52:03.372790 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.372779 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="18894e52-9d11-4a90-9a39-80d9466aadf1" containerName="registry-server" Jul 16 12:52:03.380682 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.380661 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.395383 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.395361 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5549db9dd8-z4wts"] Jul 16 12:52:03.437023 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.436992 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d385d95-ed54-4afa-ab9d-953afbb57397-trusted-ca-bundle\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.437163 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.437031 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d385d95-ed54-4afa-ab9d-953afbb57397-console-serving-cert\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.437163 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.437117 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d385d95-ed54-4afa-ab9d-953afbb57397-oauth-serving-cert\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.437312 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.437160 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d385d95-ed54-4afa-ab9d-953afbb57397-service-ca\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.437312 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.437183 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d385d95-ed54-4afa-ab9d-953afbb57397-console-config\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.437312 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.437210 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbtwc\" (UniqueName: \"kubernetes.io/projected/3d385d95-ed54-4afa-ab9d-953afbb57397-kube-api-access-cbtwc\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.437312 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.437272 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d385d95-ed54-4afa-ab9d-953afbb57397-console-oauth-config\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.538521 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.538468 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d385d95-ed54-4afa-ab9d-953afbb57397-console-serving-cert\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.538721 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.538546 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d385d95-ed54-4afa-ab9d-953afbb57397-oauth-serving-cert\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.538721 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.538585 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d385d95-ed54-4afa-ab9d-953afbb57397-service-ca\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.538721 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.538608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d385d95-ed54-4afa-ab9d-953afbb57397-console-config\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.538721 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.538629 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbtwc\" (UniqueName: \"kubernetes.io/projected/3d385d95-ed54-4afa-ab9d-953afbb57397-kube-api-access-cbtwc\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.538721 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.538664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d385d95-ed54-4afa-ab9d-953afbb57397-console-oauth-config\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.538721 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.538718 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d385d95-ed54-4afa-ab9d-953afbb57397-trusted-ca-bundle\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.539623 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.539591 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d385d95-ed54-4afa-ab9d-953afbb57397-oauth-serving-cert\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.539769 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.539642 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d385d95-ed54-4afa-ab9d-953afbb57397-console-config\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.539769 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.539650 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d385d95-ed54-4afa-ab9d-953afbb57397-trusted-ca-bundle\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.539883 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.539845 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d385d95-ed54-4afa-ab9d-953afbb57397-service-ca\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.541682 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.541662 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d385d95-ed54-4afa-ab9d-953afbb57397-console-oauth-config\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.541885 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.541865 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d385d95-ed54-4afa-ab9d-953afbb57397-console-serving-cert\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.549492 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.549468 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbtwc\" (UniqueName: \"kubernetes.io/projected/3d385d95-ed54-4afa-ab9d-953afbb57397-kube-api-access-cbtwc\") pod \"console-5549db9dd8-z4wts\" (UID: \"3d385d95-ed54-4afa-ab9d-953afbb57397\") " pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.691205 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.691118 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:03.833781 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:03.833744 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5549db9dd8-z4wts"] Jul 16 12:52:03.837118 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:52:03.837091 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d385d95_ed54_4afa_ab9d_953afbb57397.slice/crio-4d67d3b4f9a234b59c167a40b1dca5ffd71c5feeb6726518c44564e23f8902b5 WatchSource:0}: Error finding container 4d67d3b4f9a234b59c167a40b1dca5ffd71c5feeb6726518c44564e23f8902b5: Status 404 returned error can't find the container with id 4d67d3b4f9a234b59c167a40b1dca5ffd71c5feeb6726518c44564e23f8902b5 Jul 16 12:52:04.548472 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:04.548440 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5549db9dd8-z4wts" event={"ID":"3d385d95-ed54-4afa-ab9d-953afbb57397","Type":"ContainerStarted","Data":"9893ce9162ce90a63233bd25ebe2f1102cf7d0a663294f6f40efae9a90a69647"} Jul 16 12:52:04.548632 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:04.548489 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5549db9dd8-z4wts" event={"ID":"3d385d95-ed54-4afa-ab9d-953afbb57397","Type":"ContainerStarted","Data":"4d67d3b4f9a234b59c167a40b1dca5ffd71c5feeb6726518c44564e23f8902b5"} Jul 16 12:52:04.568988 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:04.568939 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5549db9dd8-z4wts" podStartSLOduration=1.5689239050000001 podStartE2EDuration="1.568923905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:52:04.56729798 +0000 UTC m=+432.562438740" watchObservedRunningTime="2026-07-16 12:52:04.568923905 +0000 UTC m=+432.564064664" Jul 16 12:52:07.720522 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:07.720482 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-8c7pg" Jul 16 12:52:07.721046 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:07.720541 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-8c7pg" Jul 16 12:52:07.744001 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:07.743975 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-8c7pg" Jul 16 12:52:08.585728 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:08.585698 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-8c7pg" Jul 16 12:52:12.837424 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:12.837386 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp"] Jul 16 12:52:12.841256 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:12.841220 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" Jul 16 12:52:12.843497 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:12.843476 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-97zh8\"" Jul 16 12:52:12.851127 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:12.851110 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp"] Jul 16 12:52:12.923205 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:12.923165 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccf6bee9-d871-4959-baa4-ec2872dde4a3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp\" (UID: \"ccf6bee9-d871-4959-baa4-ec2872dde4a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" Jul 16 12:52:12.923205 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:12.923206 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49q4w\" (UniqueName: \"kubernetes.io/projected/ccf6bee9-d871-4959-baa4-ec2872dde4a3-kube-api-access-49q4w\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp\" (UID: \"ccf6bee9-d871-4959-baa4-ec2872dde4a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" Jul 16 12:52:12.923430 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:12.923332 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccf6bee9-d871-4959-baa4-ec2872dde4a3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp\" (UID: \"ccf6bee9-d871-4959-baa4-ec2872dde4a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" Jul 16 12:52:13.024509 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.024466 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccf6bee9-d871-4959-baa4-ec2872dde4a3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp\" (UID: \"ccf6bee9-d871-4959-baa4-ec2872dde4a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" Jul 16 12:52:13.024509 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.024511 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49q4w\" (UniqueName: \"kubernetes.io/projected/ccf6bee9-d871-4959-baa4-ec2872dde4a3-kube-api-access-49q4w\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp\" (UID: \"ccf6bee9-d871-4959-baa4-ec2872dde4a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" Jul 16 12:52:13.024832 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.024591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccf6bee9-d871-4959-baa4-ec2872dde4a3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp\" (UID: \"ccf6bee9-d871-4959-baa4-ec2872dde4a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" Jul 16 12:52:13.024901 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.024865 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccf6bee9-d871-4959-baa4-ec2872dde4a3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp\" (UID: \"ccf6bee9-d871-4959-baa4-ec2872dde4a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" Jul 16 12:52:13.024976 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.024951 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccf6bee9-d871-4959-baa4-ec2872dde4a3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp\" (UID: \"ccf6bee9-d871-4959-baa4-ec2872dde4a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" Jul 16 12:52:13.034454 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.034426 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49q4w\" (UniqueName: \"kubernetes.io/projected/ccf6bee9-d871-4959-baa4-ec2872dde4a3-kube-api-access-49q4w\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp\" (UID: \"ccf6bee9-d871-4959-baa4-ec2872dde4a3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" Jul 16 12:52:13.152412 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.152331 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" Jul 16 12:52:13.286182 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.286104 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp"] Jul 16 12:52:13.288317 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:52:13.288288 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf6bee9_d871_4959_baa4_ec2872dde4a3.slice/crio-4e1702be74abcd6c45eb19c5c4221b3722fa7a0469b650be442d6a4dd0846039 WatchSource:0}: Error finding container 4e1702be74abcd6c45eb19c5c4221b3722fa7a0469b650be442d6a4dd0846039: Status 404 returned error can't find the container with id 4e1702be74abcd6c45eb19c5c4221b3722fa7a0469b650be442d6a4dd0846039 Jul 16 12:52:13.588124 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.588088 2565 generic.go:356] "Generic (PLEG): container finished" podID="ccf6bee9-d871-4959-baa4-ec2872dde4a3" containerID="f12194a70e08132ef4bd693384b16156aabdc5c7a8f6129b3eac289dc2b5684f" exitCode=0 Jul 16 12:52:13.588325 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.588167 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" event={"ID":"ccf6bee9-d871-4959-baa4-ec2872dde4a3","Type":"ContainerDied","Data":"f12194a70e08132ef4bd693384b16156aabdc5c7a8f6129b3eac289dc2b5684f"} Jul 16 12:52:13.588325 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.588212 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" event={"ID":"ccf6bee9-d871-4959-baa4-ec2872dde4a3","Type":"ContainerStarted","Data":"4e1702be74abcd6c45eb19c5c4221b3722fa7a0469b650be442d6a4dd0846039"} Jul 16 12:52:13.644878 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.644841 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh"] Jul 16 12:52:13.648404 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.648382 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" Jul 16 12:52:13.691518 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.691496 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:13.691655 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.691556 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:13.696431 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.696410 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:13.709947 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.709920 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh"] Jul 16 12:52:13.731751 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.731722 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4b68c66-4998-486a-8c72-6a0378055557-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh\" (UID: \"f4b68c66-4998-486a-8c72-6a0378055557\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" Jul 16 12:52:13.731854 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.731788 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flbj5\" (UniqueName: \"kubernetes.io/projected/f4b68c66-4998-486a-8c72-6a0378055557-kube-api-access-flbj5\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh\" (UID: \"f4b68c66-4998-486a-8c72-6a0378055557\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" Jul 16 12:52:13.731854 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.731844 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4b68c66-4998-486a-8c72-6a0378055557-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh\" (UID: \"f4b68c66-4998-486a-8c72-6a0378055557\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" Jul 16 12:52:13.832958 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.832919 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4b68c66-4998-486a-8c72-6a0378055557-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh\" (UID: \"f4b68c66-4998-486a-8c72-6a0378055557\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" Jul 16 12:52:13.833168 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.833141 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4b68c66-4998-486a-8c72-6a0378055557-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh\" (UID: \"f4b68c66-4998-486a-8c72-6a0378055557\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" Jul 16 12:52:13.833294 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.833261 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flbj5\" (UniqueName: \"kubernetes.io/projected/f4b68c66-4998-486a-8c72-6a0378055557-kube-api-access-flbj5\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh\" (UID: \"f4b68c66-4998-486a-8c72-6a0378055557\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" Jul 16 12:52:13.833453 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.833433 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4b68c66-4998-486a-8c72-6a0378055557-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh\" (UID: \"f4b68c66-4998-486a-8c72-6a0378055557\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" Jul 16 12:52:13.833525 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.833477 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4b68c66-4998-486a-8c72-6a0378055557-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh\" (UID: \"f4b68c66-4998-486a-8c72-6a0378055557\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" Jul 16 12:52:13.843105 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.843055 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flbj5\" (UniqueName: \"kubernetes.io/projected/f4b68c66-4998-486a-8c72-6a0378055557-kube-api-access-flbj5\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh\" (UID: \"f4b68c66-4998-486a-8c72-6a0378055557\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" Jul 16 12:52:13.957533 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:13.957467 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" Jul 16 12:52:14.092713 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.092692 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh"] Jul 16 12:52:14.094557 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:52:14.094500 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b68c66_4998_486a_8c72_6a0378055557.slice/crio-bff0517c70fec7941ba04ad7f82196b82264a52acfc14521f5a4a4898ca901d9 WatchSource:0}: Error finding container bff0517c70fec7941ba04ad7f82196b82264a52acfc14521f5a4a4898ca901d9: Status 404 returned error can't find the container with id bff0517c70fec7941ba04ad7f82196b82264a52acfc14521f5a4a4898ca901d9 Jul 16 12:52:14.227973 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.227946 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m"] Jul 16 12:52:14.234431 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.234388 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" Jul 16 12:52:14.241404 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.241376 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m"] Jul 16 12:52:14.338134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.338098 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efb077ca-6723-44fc-afee-ada3a8f7f000-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m\" (UID: \"efb077ca-6723-44fc-afee-ada3a8f7f000\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" Jul 16 12:52:14.338322 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.338144 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q29rn\" (UniqueName: \"kubernetes.io/projected/efb077ca-6723-44fc-afee-ada3a8f7f000-kube-api-access-q29rn\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m\" (UID: \"efb077ca-6723-44fc-afee-ada3a8f7f000\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" Jul 16 12:52:14.338322 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.338250 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efb077ca-6723-44fc-afee-ada3a8f7f000-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m\" (UID: \"efb077ca-6723-44fc-afee-ada3a8f7f000\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" Jul 16 12:52:14.438970 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.438947 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efb077ca-6723-44fc-afee-ada3a8f7f000-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m\" (UID: \"efb077ca-6723-44fc-afee-ada3a8f7f000\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" Jul 16 12:52:14.439052 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.438980 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q29rn\" (UniqueName: \"kubernetes.io/projected/efb077ca-6723-44fc-afee-ada3a8f7f000-kube-api-access-q29rn\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m\" (UID: \"efb077ca-6723-44fc-afee-ada3a8f7f000\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" Jul 16 12:52:14.439118 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.439105 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efb077ca-6723-44fc-afee-ada3a8f7f000-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m\" (UID: \"efb077ca-6723-44fc-afee-ada3a8f7f000\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" Jul 16 12:52:14.439316 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.439296 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efb077ca-6723-44fc-afee-ada3a8f7f000-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m\" (UID: \"efb077ca-6723-44fc-afee-ada3a8f7f000\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" Jul 16 12:52:14.439414 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.439395 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efb077ca-6723-44fc-afee-ada3a8f7f000-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m\" (UID: \"efb077ca-6723-44fc-afee-ada3a8f7f000\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" Jul 16 12:52:14.447839 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.447814 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q29rn\" (UniqueName: \"kubernetes.io/projected/efb077ca-6723-44fc-afee-ada3a8f7f000-kube-api-access-q29rn\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m\" (UID: \"efb077ca-6723-44fc-afee-ada3a8f7f000\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" Jul 16 12:52:14.557785 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.557759 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" Jul 16 12:52:14.594923 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.594890 2565 generic.go:356] "Generic (PLEG): container finished" podID="ccf6bee9-d871-4959-baa4-ec2872dde4a3" containerID="50660b12eb407ddca8f90d42f940763ef413e1b5c27bf2ba299a5c7fd7c19ad2" exitCode=0 Jul 16 12:52:14.595068 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.594937 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" event={"ID":"ccf6bee9-d871-4959-baa4-ec2872dde4a3","Type":"ContainerDied","Data":"50660b12eb407ddca8f90d42f940763ef413e1b5c27bf2ba299a5c7fd7c19ad2"} Jul 16 12:52:14.596812 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.596782 2565 generic.go:356] "Generic (PLEG): container finished" podID="f4b68c66-4998-486a-8c72-6a0378055557" containerID="bd2dfd0092a713fa02fe0f5e75435bb2cb258898ed09f52f5712d06ac05762a1" exitCode=0 Jul 16 12:52:14.596893 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.596870 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" event={"ID":"f4b68c66-4998-486a-8c72-6a0378055557","Type":"ContainerDied","Data":"bd2dfd0092a713fa02fe0f5e75435bb2cb258898ed09f52f5712d06ac05762a1"} Jul 16 12:52:14.596943 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.596910 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" event={"ID":"f4b68c66-4998-486a-8c72-6a0378055557","Type":"ContainerStarted","Data":"bff0517c70fec7941ba04ad7f82196b82264a52acfc14521f5a4a4898ca901d9"} Jul 16 12:52:14.601972 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.601950 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5549db9dd8-z4wts" Jul 16 12:52:14.639793 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.639763 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh"] Jul 16 12:52:14.644476 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.644454 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" Jul 16 12:52:14.656401 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.656271 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh"] Jul 16 12:52:14.691362 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.691289 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9bb975cf7-qgz42"] Jul 16 12:52:14.718705 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.718670 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m"] Jul 16 12:52:14.721001 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:52:14.720962 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb077ca_6723_44fc_afee_ada3a8f7f000.slice/crio-a5330db179e52f3741ceb804cad7084de608251ecc021d911cdf655668d44fc4 WatchSource:0}: Error finding container a5330db179e52f3741ceb804cad7084de608251ecc021d911cdf655668d44fc4: Status 404 returned error can't find the container with id a5330db179e52f3741ceb804cad7084de608251ecc021d911cdf655668d44fc4 Jul 16 12:52:14.741294 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.741259 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00257efc-a528-46ae-8755-3ced72e384be-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh\" (UID: \"00257efc-a528-46ae-8755-3ced72e384be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" Jul 16 12:52:14.741430 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.741312 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00257efc-a528-46ae-8755-3ced72e384be-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh\" (UID: \"00257efc-a528-46ae-8755-3ced72e384be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" Jul 16 12:52:14.741430 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.741415 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-654rl\" (UniqueName: \"kubernetes.io/projected/00257efc-a528-46ae-8755-3ced72e384be-kube-api-access-654rl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh\" (UID: \"00257efc-a528-46ae-8755-3ced72e384be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" Jul 16 12:52:14.842667 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.842628 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-654rl\" (UniqueName: \"kubernetes.io/projected/00257efc-a528-46ae-8755-3ced72e384be-kube-api-access-654rl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh\" (UID: \"00257efc-a528-46ae-8755-3ced72e384be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" Jul 16 12:52:14.842812 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.842730 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00257efc-a528-46ae-8755-3ced72e384be-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh\" (UID: \"00257efc-a528-46ae-8755-3ced72e384be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" Jul 16 12:52:14.842812 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.842759 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00257efc-a528-46ae-8755-3ced72e384be-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh\" (UID: \"00257efc-a528-46ae-8755-3ced72e384be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" Jul 16 12:52:14.843113 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.843088 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00257efc-a528-46ae-8755-3ced72e384be-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh\" (UID: \"00257efc-a528-46ae-8755-3ced72e384be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" Jul 16 12:52:14.843647 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.843118 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00257efc-a528-46ae-8755-3ced72e384be-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh\" (UID: \"00257efc-a528-46ae-8755-3ced72e384be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" Jul 16 12:52:14.858803 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.858770 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-654rl\" (UniqueName: \"kubernetes.io/projected/00257efc-a528-46ae-8755-3ced72e384be-kube-api-access-654rl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh\" (UID: \"00257efc-a528-46ae-8755-3ced72e384be\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" Jul 16 12:52:14.957389 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:14.957316 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" Jul 16 12:52:15.095182 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:15.095155 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh"] Jul 16 12:52:15.097805 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:52:15.097777 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00257efc_a528_46ae_8755_3ced72e384be.slice/crio-ff64e2aa19d0372f66b332ce5f477d2812fd7a9e05f48bef40f9972f8675b154 WatchSource:0}: Error finding container ff64e2aa19d0372f66b332ce5f477d2812fd7a9e05f48bef40f9972f8675b154: Status 404 returned error can't find the container with id ff64e2aa19d0372f66b332ce5f477d2812fd7a9e05f48bef40f9972f8675b154 Jul 16 12:52:15.603408 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:15.603375 2565 generic.go:356] "Generic (PLEG): container finished" podID="efb077ca-6723-44fc-afee-ada3a8f7f000" containerID="bb63e388381e0690b4a7da11cdbb63799f70a36ad38fec6a6a414b523d5ca7b4" exitCode=0 Jul 16 12:52:15.603580 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:15.603421 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" event={"ID":"efb077ca-6723-44fc-afee-ada3a8f7f000","Type":"ContainerDied","Data":"bb63e388381e0690b4a7da11cdbb63799f70a36ad38fec6a6a414b523d5ca7b4"} Jul 16 12:52:15.603580 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:15.603470 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" event={"ID":"efb077ca-6723-44fc-afee-ada3a8f7f000","Type":"ContainerStarted","Data":"a5330db179e52f3741ceb804cad7084de608251ecc021d911cdf655668d44fc4"} Jul 16 12:52:15.605087 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:15.605056 2565 generic.go:356] "Generic (PLEG): container finished" podID="00257efc-a528-46ae-8755-3ced72e384be" containerID="37af4a063023abe37fb87566d7c38cd6f44e35f77742244562f4161063828c29" exitCode=0 Jul 16 12:52:15.605204 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:15.605119 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" event={"ID":"00257efc-a528-46ae-8755-3ced72e384be","Type":"ContainerDied","Data":"37af4a063023abe37fb87566d7c38cd6f44e35f77742244562f4161063828c29"} Jul 16 12:52:15.605204 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:15.605156 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" event={"ID":"00257efc-a528-46ae-8755-3ced72e384be","Type":"ContainerStarted","Data":"ff64e2aa19d0372f66b332ce5f477d2812fd7a9e05f48bef40f9972f8675b154"} Jul 16 12:52:15.607441 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:15.607417 2565 generic.go:356] "Generic (PLEG): container finished" podID="ccf6bee9-d871-4959-baa4-ec2872dde4a3" containerID="52ce6a523c8202eb4170e61e6b7249aa2a1638da7efe87f8f0be69f6598cf98a" exitCode=0 Jul 16 12:52:15.607557 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:15.607509 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" event={"ID":"ccf6bee9-d871-4959-baa4-ec2872dde4a3","Type":"ContainerDied","Data":"52ce6a523c8202eb4170e61e6b7249aa2a1638da7efe87f8f0be69f6598cf98a"} Jul 16 12:52:15.609359 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:15.609340 2565 generic.go:356] "Generic (PLEG): container finished" podID="f4b68c66-4998-486a-8c72-6a0378055557" containerID="f40f52758d9026cf90a21aed6f29401b338e2e7dd060fc71e08e0d5a6a92c9f0" exitCode=0 Jul 16 12:52:15.609457 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:15.609421 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" event={"ID":"f4b68c66-4998-486a-8c72-6a0378055557","Type":"ContainerDied","Data":"f40f52758d9026cf90a21aed6f29401b338e2e7dd060fc71e08e0d5a6a92c9f0"} Jul 16 12:52:16.617701 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:16.617667 2565 generic.go:356] "Generic (PLEG): container finished" podID="f4b68c66-4998-486a-8c72-6a0378055557" containerID="34ba3f105a50dfa4cc7bca3275d035fe22cc11b1966cbc848c1ced2dd593a2ae" exitCode=0 Jul 16 12:52:16.618088 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:16.617754 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" event={"ID":"f4b68c66-4998-486a-8c72-6a0378055557","Type":"ContainerDied","Data":"34ba3f105a50dfa4cc7bca3275d035fe22cc11b1966cbc848c1ced2dd593a2ae"} Jul 16 12:52:16.619692 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:16.619668 2565 generic.go:356] "Generic (PLEG): container finished" podID="00257efc-a528-46ae-8755-3ced72e384be" containerID="7bfd53bf013658216f61050a658a740681e521758b69644b42e7234904f9a733" exitCode=0 Jul 16 12:52:16.619831 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:16.619745 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" event={"ID":"00257efc-a528-46ae-8755-3ced72e384be","Type":"ContainerDied","Data":"7bfd53bf013658216f61050a658a740681e521758b69644b42e7234904f9a733"} Jul 16 12:52:16.757024 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:16.756990 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" Jul 16 12:52:16.862467 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:16.862439 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccf6bee9-d871-4959-baa4-ec2872dde4a3-util\") pod \"ccf6bee9-d871-4959-baa4-ec2872dde4a3\" (UID: \"ccf6bee9-d871-4959-baa4-ec2872dde4a3\") " Jul 16 12:52:16.862572 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:16.862505 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccf6bee9-d871-4959-baa4-ec2872dde4a3-bundle\") pod \"ccf6bee9-d871-4959-baa4-ec2872dde4a3\" (UID: \"ccf6bee9-d871-4959-baa4-ec2872dde4a3\") " Jul 16 12:52:16.862626 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:16.862583 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49q4w\" (UniqueName: \"kubernetes.io/projected/ccf6bee9-d871-4959-baa4-ec2872dde4a3-kube-api-access-49q4w\") pod \"ccf6bee9-d871-4959-baa4-ec2872dde4a3\" (UID: \"ccf6bee9-d871-4959-baa4-ec2872dde4a3\") " Jul 16 12:52:16.862993 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:16.862957 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf6bee9-d871-4959-baa4-ec2872dde4a3-bundle" (OuterVolumeSpecName: "bundle") pod "ccf6bee9-d871-4959-baa4-ec2872dde4a3" (UID: "ccf6bee9-d871-4959-baa4-ec2872dde4a3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:52:16.864629 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:16.864606 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf6bee9-d871-4959-baa4-ec2872dde4a3-kube-api-access-49q4w" (OuterVolumeSpecName: "kube-api-access-49q4w") pod "ccf6bee9-d871-4959-baa4-ec2872dde4a3" (UID: "ccf6bee9-d871-4959-baa4-ec2872dde4a3"). InnerVolumeSpecName "kube-api-access-49q4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:52:16.867790 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:16.867769 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf6bee9-d871-4959-baa4-ec2872dde4a3-util" (OuterVolumeSpecName: "util") pod "ccf6bee9-d871-4959-baa4-ec2872dde4a3" (UID: "ccf6bee9-d871-4959-baa4-ec2872dde4a3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:52:16.963867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:16.963799 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccf6bee9-d871-4959-baa4-ec2872dde4a3-bundle\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:16.963867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:16.963826 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49q4w\" (UniqueName: \"kubernetes.io/projected/ccf6bee9-d871-4959-baa4-ec2872dde4a3-kube-api-access-49q4w\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:16.963867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:16.963838 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccf6bee9-d871-4959-baa4-ec2872dde4a3-util\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:17.626687 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.626649 2565 generic.go:356] "Generic (PLEG): container finished" podID="efb077ca-6723-44fc-afee-ada3a8f7f000" containerID="ca09a096e536fad767bba87b43174ddb701cf1ab3249bcec813454ae47e45606" exitCode=0 Jul 16 12:52:17.627139 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.626740 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" event={"ID":"efb077ca-6723-44fc-afee-ada3a8f7f000","Type":"ContainerDied","Data":"ca09a096e536fad767bba87b43174ddb701cf1ab3249bcec813454ae47e45606"} Jul 16 12:52:17.628957 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.628932 2565 generic.go:356] "Generic (PLEG): container finished" podID="00257efc-a528-46ae-8755-3ced72e384be" containerID="f34a627745793181e872675b06204d3e64e669029d207f63719da70e72e2b4f8" exitCode=0 Jul 16 12:52:17.629081 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.629022 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" event={"ID":"00257efc-a528-46ae-8755-3ced72e384be","Type":"ContainerDied","Data":"f34a627745793181e872675b06204d3e64e669029d207f63719da70e72e2b4f8"} Jul 16 12:52:17.630741 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.630719 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" Jul 16 12:52:17.630826 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.630726 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp" event={"ID":"ccf6bee9-d871-4959-baa4-ec2872dde4a3","Type":"ContainerDied","Data":"4e1702be74abcd6c45eb19c5c4221b3722fa7a0469b650be442d6a4dd0846039"} Jul 16 12:52:17.630826 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.630782 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e1702be74abcd6c45eb19c5c4221b3722fa7a0469b650be442d6a4dd0846039" Jul 16 12:52:17.773527 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.773499 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" Jul 16 12:52:17.870507 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.870476 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flbj5\" (UniqueName: \"kubernetes.io/projected/f4b68c66-4998-486a-8c72-6a0378055557-kube-api-access-flbj5\") pod \"f4b68c66-4998-486a-8c72-6a0378055557\" (UID: \"f4b68c66-4998-486a-8c72-6a0378055557\") " Jul 16 12:52:17.870633 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.870524 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4b68c66-4998-486a-8c72-6a0378055557-bundle\") pod \"f4b68c66-4998-486a-8c72-6a0378055557\" (UID: \"f4b68c66-4998-486a-8c72-6a0378055557\") " Jul 16 12:52:17.870713 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.870647 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4b68c66-4998-486a-8c72-6a0378055557-util\") pod \"f4b68c66-4998-486a-8c72-6a0378055557\" (UID: \"f4b68c66-4998-486a-8c72-6a0378055557\") " Jul 16 12:52:17.871015 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.870992 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b68c66-4998-486a-8c72-6a0378055557-bundle" (OuterVolumeSpecName: "bundle") pod "f4b68c66-4998-486a-8c72-6a0378055557" (UID: "f4b68c66-4998-486a-8c72-6a0378055557"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:52:17.872712 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.872689 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b68c66-4998-486a-8c72-6a0378055557-kube-api-access-flbj5" (OuterVolumeSpecName: "kube-api-access-flbj5") pod "f4b68c66-4998-486a-8c72-6a0378055557" (UID: "f4b68c66-4998-486a-8c72-6a0378055557"). InnerVolumeSpecName "kube-api-access-flbj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:52:17.875834 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.875815 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b68c66-4998-486a-8c72-6a0378055557-util" (OuterVolumeSpecName: "util") pod "f4b68c66-4998-486a-8c72-6a0378055557" (UID: "f4b68c66-4998-486a-8c72-6a0378055557"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:52:17.972299 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.972213 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flbj5\" (UniqueName: \"kubernetes.io/projected/f4b68c66-4998-486a-8c72-6a0378055557-kube-api-access-flbj5\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:17.972299 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.972265 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4b68c66-4998-486a-8c72-6a0378055557-bundle\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:17.972299 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:17.972276 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4b68c66-4998-486a-8c72-6a0378055557-util\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:18.638523 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:18.638493 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" Jul 16 12:52:18.638919 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:18.638522 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh" event={"ID":"f4b68c66-4998-486a-8c72-6a0378055557","Type":"ContainerDied","Data":"bff0517c70fec7941ba04ad7f82196b82264a52acfc14521f5a4a4898ca901d9"} Jul 16 12:52:18.638919 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:18.638577 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bff0517c70fec7941ba04ad7f82196b82264a52acfc14521f5a4a4898ca901d9" Jul 16 12:52:18.640768 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:18.640745 2565 generic.go:356] "Generic (PLEG): container finished" podID="efb077ca-6723-44fc-afee-ada3a8f7f000" containerID="d59cbd3ada50ea2a4740ee74d7155fab4e125dca48c10cf3543b952b9ae7b972" exitCode=0 Jul 16 12:52:18.640902 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:18.640829 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" event={"ID":"efb077ca-6723-44fc-afee-ada3a8f7f000","Type":"ContainerDied","Data":"d59cbd3ada50ea2a4740ee74d7155fab4e125dca48c10cf3543b952b9ae7b972"} Jul 16 12:52:18.774286 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:18.774261 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" Jul 16 12:52:18.879140 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:18.879066 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00257efc-a528-46ae-8755-3ced72e384be-bundle\") pod \"00257efc-a528-46ae-8755-3ced72e384be\" (UID: \"00257efc-a528-46ae-8755-3ced72e384be\") " Jul 16 12:52:18.879140 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:18.879105 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00257efc-a528-46ae-8755-3ced72e384be-util\") pod \"00257efc-a528-46ae-8755-3ced72e384be\" (UID: \"00257efc-a528-46ae-8755-3ced72e384be\") " Jul 16 12:52:18.879383 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:18.879209 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-654rl\" (UniqueName: \"kubernetes.io/projected/00257efc-a528-46ae-8755-3ced72e384be-kube-api-access-654rl\") pod \"00257efc-a528-46ae-8755-3ced72e384be\" (UID: \"00257efc-a528-46ae-8755-3ced72e384be\") " Jul 16 12:52:18.879678 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:18.879650 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00257efc-a528-46ae-8755-3ced72e384be-bundle" (OuterVolumeSpecName: "bundle") pod "00257efc-a528-46ae-8755-3ced72e384be" (UID: "00257efc-a528-46ae-8755-3ced72e384be"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:52:18.881490 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:18.881463 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00257efc-a528-46ae-8755-3ced72e384be-kube-api-access-654rl" (OuterVolumeSpecName: "kube-api-access-654rl") pod "00257efc-a528-46ae-8755-3ced72e384be" (UID: "00257efc-a528-46ae-8755-3ced72e384be"). InnerVolumeSpecName "kube-api-access-654rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:52:18.884645 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:18.884625 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00257efc-a528-46ae-8755-3ced72e384be-util" (OuterVolumeSpecName: "util") pod "00257efc-a528-46ae-8755-3ced72e384be" (UID: "00257efc-a528-46ae-8755-3ced72e384be"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:52:18.980307 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:18.980283 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-654rl\" (UniqueName: \"kubernetes.io/projected/00257efc-a528-46ae-8755-3ced72e384be-kube-api-access-654rl\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:18.980307 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:18.980304 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00257efc-a528-46ae-8755-3ced72e384be-bundle\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:18.980414 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:18.980312 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00257efc-a528-46ae-8755-3ced72e384be-util\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:19.648974 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:19.648938 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" event={"ID":"00257efc-a528-46ae-8755-3ced72e384be","Type":"ContainerDied","Data":"ff64e2aa19d0372f66b332ce5f477d2812fd7a9e05f48bef40f9972f8675b154"} Jul 16 12:52:19.648974 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:19.648966 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh" Jul 16 12:52:19.649472 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:19.648988 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff64e2aa19d0372f66b332ce5f477d2812fd7a9e05f48bef40f9972f8675b154" Jul 16 12:52:19.792341 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:19.792317 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" Jul 16 12:52:19.888292 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:19.888250 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q29rn\" (UniqueName: \"kubernetes.io/projected/efb077ca-6723-44fc-afee-ada3a8f7f000-kube-api-access-q29rn\") pod \"efb077ca-6723-44fc-afee-ada3a8f7f000\" (UID: \"efb077ca-6723-44fc-afee-ada3a8f7f000\") " Jul 16 12:52:19.888470 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:19.888315 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efb077ca-6723-44fc-afee-ada3a8f7f000-util\") pod \"efb077ca-6723-44fc-afee-ada3a8f7f000\" (UID: \"efb077ca-6723-44fc-afee-ada3a8f7f000\") " Jul 16 12:52:19.888470 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:19.888358 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efb077ca-6723-44fc-afee-ada3a8f7f000-bundle\") pod \"efb077ca-6723-44fc-afee-ada3a8f7f000\" (UID: \"efb077ca-6723-44fc-afee-ada3a8f7f000\") " Jul 16 12:52:19.888985 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:19.888959 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb077ca-6723-44fc-afee-ada3a8f7f000-bundle" (OuterVolumeSpecName: "bundle") pod "efb077ca-6723-44fc-afee-ada3a8f7f000" (UID: "efb077ca-6723-44fc-afee-ada3a8f7f000"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:52:19.890506 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:19.890471 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb077ca-6723-44fc-afee-ada3a8f7f000-kube-api-access-q29rn" (OuterVolumeSpecName: "kube-api-access-q29rn") pod "efb077ca-6723-44fc-afee-ada3a8f7f000" (UID: "efb077ca-6723-44fc-afee-ada3a8f7f000"). InnerVolumeSpecName "kube-api-access-q29rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:52:19.893825 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:19.893801 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb077ca-6723-44fc-afee-ada3a8f7f000-util" (OuterVolumeSpecName: "util") pod "efb077ca-6723-44fc-afee-ada3a8f7f000" (UID: "efb077ca-6723-44fc-afee-ada3a8f7f000"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:52:19.989937 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:19.989872 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q29rn\" (UniqueName: \"kubernetes.io/projected/efb077ca-6723-44fc-afee-ada3a8f7f000-kube-api-access-q29rn\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:19.989937 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:19.989897 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efb077ca-6723-44fc-afee-ada3a8f7f000-util\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:19.989937 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:19.989906 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efb077ca-6723-44fc-afee-ada3a8f7f000-bundle\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:20.655944 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:20.655917 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" Jul 16 12:52:20.656403 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:20.655950 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m" event={"ID":"efb077ca-6723-44fc-afee-ada3a8f7f000","Type":"ContainerDied","Data":"a5330db179e52f3741ceb804cad7084de608251ecc021d911cdf655668d44fc4"} Jul 16 12:52:20.656403 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:20.655992 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5330db179e52f3741ceb804cad7084de608251ecc021d911cdf655668d44fc4" Jul 16 12:52:25.933337 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933300 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbjqw"] Jul 16 12:52:25.933696 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933674 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb077ca-6723-44fc-afee-ada3a8f7f000" containerName="extract" Jul 16 12:52:25.933696 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933685 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb077ca-6723-44fc-afee-ada3a8f7f000" containerName="extract" Jul 16 12:52:25.933781 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933704 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b68c66-4998-486a-8c72-6a0378055557" containerName="util" Jul 16 12:52:25.933781 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933710 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b68c66-4998-486a-8c72-6a0378055557" containerName="util" Jul 16 12:52:25.933781 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933718 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb077ca-6723-44fc-afee-ada3a8f7f000" containerName="pull" Jul 16 12:52:25.933781 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933724 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb077ca-6723-44fc-afee-ada3a8f7f000" containerName="pull" Jul 16 12:52:25.933781 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933732 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccf6bee9-d871-4959-baa4-ec2872dde4a3" containerName="extract" Jul 16 12:52:25.933781 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933737 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccf6bee9-d871-4959-baa4-ec2872dde4a3" containerName="extract" Jul 16 12:52:25.933781 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933743 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b68c66-4998-486a-8c72-6a0378055557" containerName="extract" Jul 16 12:52:25.933781 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933749 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b68c66-4998-486a-8c72-6a0378055557" containerName="extract" Jul 16 12:52:25.933781 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933757 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00257efc-a528-46ae-8755-3ced72e384be" containerName="pull" Jul 16 12:52:25.933781 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933762 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="00257efc-a528-46ae-8755-3ced72e384be" containerName="pull" Jul 16 12:52:25.933781 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933774 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb077ca-6723-44fc-afee-ada3a8f7f000" containerName="util" Jul 16 12:52:25.933781 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933779 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb077ca-6723-44fc-afee-ada3a8f7f000" containerName="util" Jul 16 12:52:25.933781 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933784 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00257efc-a528-46ae-8755-3ced72e384be" containerName="util" Jul 16 12:52:25.934134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933790 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="00257efc-a528-46ae-8755-3ced72e384be" containerName="util" Jul 16 12:52:25.934134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933796 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00257efc-a528-46ae-8755-3ced72e384be" containerName="extract" Jul 16 12:52:25.934134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933802 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="00257efc-a528-46ae-8755-3ced72e384be" containerName="extract" Jul 16 12:52:25.934134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933808 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccf6bee9-d871-4959-baa4-ec2872dde4a3" containerName="pull" Jul 16 12:52:25.934134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933814 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccf6bee9-d871-4959-baa4-ec2872dde4a3" containerName="pull" Jul 16 12:52:25.934134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933822 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccf6bee9-d871-4959-baa4-ec2872dde4a3" containerName="util" Jul 16 12:52:25.934134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933827 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccf6bee9-d871-4959-baa4-ec2872dde4a3" containerName="util" Jul 16 12:52:25.934134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933833 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b68c66-4998-486a-8c72-6a0378055557" containerName="pull" Jul 16 12:52:25.934134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933838 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b68c66-4998-486a-8c72-6a0378055557" containerName="pull" Jul 16 12:52:25.934134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933885 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccf6bee9-d871-4959-baa4-ec2872dde4a3" containerName="extract" Jul 16 12:52:25.934134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933893 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb077ca-6723-44fc-afee-ada3a8f7f000" containerName="extract" Jul 16 12:52:25.934134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933901 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="00257efc-a528-46ae-8755-3ced72e384be" containerName="extract" Jul 16 12:52:25.934134 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.933907 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4b68c66-4998-486a-8c72-6a0378055557" containerName="extract" Jul 16 12:52:25.937117 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.937097 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbjqw" Jul 16 12:52:25.939650 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.939630 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 16 12:52:25.939818 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.939804 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zb68j\"" Jul 16 12:52:25.950342 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:25.950321 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbjqw"] Jul 16 12:52:26.043995 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:26.043963 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgkhb\" (UniqueName: \"kubernetes.io/projected/a7947a67-b83e-4e01-941c-56704482046b-kube-api-access-jgkhb\") pod \"dns-operator-controller-manager-864dd78c4c-dbjqw\" (UID: \"a7947a67-b83e-4e01-941c-56704482046b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbjqw" Jul 16 12:52:26.145304 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:26.145269 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkhb\" (UniqueName: \"kubernetes.io/projected/a7947a67-b83e-4e01-941c-56704482046b-kube-api-access-jgkhb\") pod \"dns-operator-controller-manager-864dd78c4c-dbjqw\" (UID: \"a7947a67-b83e-4e01-941c-56704482046b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbjqw" Jul 16 12:52:26.160173 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:26.160143 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgkhb\" (UniqueName: \"kubernetes.io/projected/a7947a67-b83e-4e01-941c-56704482046b-kube-api-access-jgkhb\") pod \"dns-operator-controller-manager-864dd78c4c-dbjqw\" (UID: \"a7947a67-b83e-4e01-941c-56704482046b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbjqw" Jul 16 12:52:26.247020 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:26.246935 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbjqw" Jul 16 12:52:26.396199 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:26.396176 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbjqw"] Jul 16 12:52:26.397668 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:52:26.397641 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7947a67_b83e_4e01_941c_56704482046b.slice/crio-c733287c330bd527dc906e83f3590f17bcfa29cb969406fbfbbc4eb8c373e9fb WatchSource:0}: Error finding container c733287c330bd527dc906e83f3590f17bcfa29cb969406fbfbbc4eb8c373e9fb: Status 404 returned error can't find the container with id c733287c330bd527dc906e83f3590f17bcfa29cb969406fbfbbc4eb8c373e9fb Jul 16 12:52:26.691922 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:26.691890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbjqw" event={"ID":"a7947a67-b83e-4e01-941c-56704482046b","Type":"ContainerStarted","Data":"c733287c330bd527dc906e83f3590f17bcfa29cb969406fbfbbc4eb8c373e9fb"} Jul 16 12:52:29.375738 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:29.375703 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg"] Jul 16 12:52:29.379136 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:29.379120 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" Jul 16 12:52:29.381659 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:29.381629 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fjfn2\"" Jul 16 12:52:29.398693 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:29.398672 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg"] Jul 16 12:52:29.473447 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:29.473409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8xz5\" (UniqueName: \"kubernetes.io/projected/a53ed1d2-ef85-4ae3-aecd-c85c17661657-kube-api-access-m8xz5\") pod \"limitador-operator-controller-manager-859d7b78bd-ltfbg\" (UID: \"a53ed1d2-ef85-4ae3-aecd-c85c17661657\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" Jul 16 12:52:29.574600 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:29.574566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8xz5\" (UniqueName: \"kubernetes.io/projected/a53ed1d2-ef85-4ae3-aecd-c85c17661657-kube-api-access-m8xz5\") pod \"limitador-operator-controller-manager-859d7b78bd-ltfbg\" (UID: \"a53ed1d2-ef85-4ae3-aecd-c85c17661657\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" Jul 16 12:52:29.587280 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:29.587250 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8xz5\" (UniqueName: \"kubernetes.io/projected/a53ed1d2-ef85-4ae3-aecd-c85c17661657-kube-api-access-m8xz5\") pod \"limitador-operator-controller-manager-859d7b78bd-ltfbg\" (UID: \"a53ed1d2-ef85-4ae3-aecd-c85c17661657\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" Jul 16 12:52:29.689088 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:29.689014 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" Jul 16 12:52:29.712010 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:29.711975 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbjqw" event={"ID":"a7947a67-b83e-4e01-941c-56704482046b","Type":"ContainerStarted","Data":"9eac20c82936bbfd6fe6eef1953a4912c502a13d32549690c458a1ddc4a79137"} Jul 16 12:52:29.712158 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:29.712060 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbjqw" Jul 16 12:52:29.745839 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:29.745640 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbjqw" podStartSLOduration=2.499242543 podStartE2EDuration="4.745620202s" totalImagesPullingTime="2.246377659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:52:29.744777855 +0000 UTC m=+457.739918647" watchObservedRunningTime="2026-07-16 12:52:29.745620202 +0000 UTC m=+457.740760963" Jul 16 12:52:29.837805 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:29.837778 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg"] Jul 16 12:52:29.840311 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:52:29.840282 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53ed1d2_ef85_4ae3_aecd_c85c17661657.slice/crio-0192d15f5415f134be45084552461654369be51422437d1ca605179aa2868687 WatchSource:0}: Error finding container 0192d15f5415f134be45084552461654369be51422437d1ca605179aa2868687: Status 404 returned error can't find the container with id 0192d15f5415f134be45084552461654369be51422437d1ca605179aa2868687 Jul 16 12:52:30.718148 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:30.718112 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" event={"ID":"a53ed1d2-ef85-4ae3-aecd-c85c17661657","Type":"ContainerStarted","Data":"0192d15f5415f134be45084552461654369be51422437d1ca605179aa2868687"} Jul 16 12:52:33.736817 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:33.736778 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" event={"ID":"a53ed1d2-ef85-4ae3-aecd-c85c17661657","Type":"ContainerStarted","Data":"cb0209abb79c93f3acf3e5e2c8dd763b351cac82186878b025e309613b060ddd"} Jul 16 12:52:33.737185 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:33.736943 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" Jul 16 12:52:33.757292 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:33.757244 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" podStartSLOduration=1.712404205 podStartE2EDuration="4.757211626s" totalImagesPullingTime="3.044807421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:52:33.755037599 +0000 UTC m=+461.750178379" watchObservedRunningTime="2026-07-16 12:52:33.757211626 +0000 UTC m=+461.752352385" Jul 16 12:52:37.891723 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:37.891687 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-w9d5v"] Jul 16 12:52:37.896891 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:37.896873 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w9d5v" Jul 16 12:52:37.900065 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:37.900049 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-66sjr\"" Jul 16 12:52:37.911509 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:37.911484 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-w9d5v"] Jul 16 12:52:37.950655 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:37.950615 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvmrz\" (UniqueName: \"kubernetes.io/projected/58dca71d-bb6d-406f-9699-fe45c5d9bbed-kube-api-access-fvmrz\") pod \"authorino-operator-75cc8d5cb7-w9d5v\" (UID: \"58dca71d-bb6d-406f-9699-fe45c5d9bbed\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-w9d5v" Jul 16 12:52:38.051868 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:38.051835 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvmrz\" (UniqueName: \"kubernetes.io/projected/58dca71d-bb6d-406f-9699-fe45c5d9bbed-kube-api-access-fvmrz\") pod \"authorino-operator-75cc8d5cb7-w9d5v\" (UID: \"58dca71d-bb6d-406f-9699-fe45c5d9bbed\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-w9d5v" Jul 16 12:52:38.062058 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:38.062025 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvmrz\" (UniqueName: \"kubernetes.io/projected/58dca71d-bb6d-406f-9699-fe45c5d9bbed-kube-api-access-fvmrz\") pod \"authorino-operator-75cc8d5cb7-w9d5v\" (UID: \"58dca71d-bb6d-406f-9699-fe45c5d9bbed\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-w9d5v" Jul 16 12:52:38.206125 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:38.206040 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w9d5v" Jul 16 12:52:38.346155 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:38.346131 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-w9d5v"] Jul 16 12:52:38.348087 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:52:38.348055 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58dca71d_bb6d_406f_9699_fe45c5d9bbed.slice/crio-4cd8db31da980c3b380dbba623c6811f83784888abac2b33669c916683564973 WatchSource:0}: Error finding container 4cd8db31da980c3b380dbba623c6811f83784888abac2b33669c916683564973: Status 404 returned error can't find the container with id 4cd8db31da980c3b380dbba623c6811f83784888abac2b33669c916683564973 Jul 16 12:52:38.767997 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:38.767961 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w9d5v" event={"ID":"58dca71d-bb6d-406f-9699-fe45c5d9bbed","Type":"ContainerStarted","Data":"4cd8db31da980c3b380dbba623c6811f83784888abac2b33669c916683564973"} Jul 16 12:52:39.721026 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:39.720985 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9bb975cf7-qgz42" podUID="e883afe2-3f49-48ab-9759-974238eb7f69" containerName="console" containerID="cri-o://533ef659f6f67b02db1de90fbfe39920c958f4c609f96c84852937be39c62a7a" gracePeriod=15 Jul 16 12:52:39.971632 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:39.971568 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:52:40.068186 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.068157 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m24k\" (UniqueName: \"kubernetes.io/projected/e883afe2-3f49-48ab-9759-974238eb7f69-kube-api-access-5m24k\") pod \"e883afe2-3f49-48ab-9759-974238eb7f69\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " Jul 16 12:52:40.068384 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.068266 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e883afe2-3f49-48ab-9759-974238eb7f69-console-oauth-config\") pod \"e883afe2-3f49-48ab-9759-974238eb7f69\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " Jul 16 12:52:40.068384 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.068300 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e883afe2-3f49-48ab-9759-974238eb7f69-console-serving-cert\") pod \"e883afe2-3f49-48ab-9759-974238eb7f69\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " Jul 16 12:52:40.068384 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.068343 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-trusted-ca-bundle\") pod \"e883afe2-3f49-48ab-9759-974238eb7f69\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " Jul 16 12:52:40.068384 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.068376 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-console-config\") pod \"e883afe2-3f49-48ab-9759-974238eb7f69\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " Jul 16 12:52:40.068601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.068423 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-service-ca\") pod \"e883afe2-3f49-48ab-9759-974238eb7f69\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " Jul 16 12:52:40.068601 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.068459 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-oauth-serving-cert\") pod \"e883afe2-3f49-48ab-9759-974238eb7f69\" (UID: \"e883afe2-3f49-48ab-9759-974238eb7f69\") " Jul 16 12:52:40.068818 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.068785 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e883afe2-3f49-48ab-9759-974238eb7f69" (UID: "e883afe2-3f49-48ab-9759-974238eb7f69"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:52:40.068951 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.068824 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-console-config" (OuterVolumeSpecName: "console-config") pod "e883afe2-3f49-48ab-9759-974238eb7f69" (UID: "e883afe2-3f49-48ab-9759-974238eb7f69"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:52:40.069183 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.069014 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e883afe2-3f49-48ab-9759-974238eb7f69" (UID: "e883afe2-3f49-48ab-9759-974238eb7f69"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:52:40.069268 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.069208 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-service-ca" (OuterVolumeSpecName: "service-ca") pod "e883afe2-3f49-48ab-9759-974238eb7f69" (UID: "e883afe2-3f49-48ab-9759-974238eb7f69"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:52:40.070607 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.070580 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e883afe2-3f49-48ab-9759-974238eb7f69-kube-api-access-5m24k" (OuterVolumeSpecName: "kube-api-access-5m24k") pod "e883afe2-3f49-48ab-9759-974238eb7f69" (UID: "e883afe2-3f49-48ab-9759-974238eb7f69"). InnerVolumeSpecName "kube-api-access-5m24k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:52:40.070707 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.070620 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e883afe2-3f49-48ab-9759-974238eb7f69-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e883afe2-3f49-48ab-9759-974238eb7f69" (UID: "e883afe2-3f49-48ab-9759-974238eb7f69"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:52:40.070707 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.070638 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e883afe2-3f49-48ab-9759-974238eb7f69-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e883afe2-3f49-48ab-9759-974238eb7f69" (UID: "e883afe2-3f49-48ab-9759-974238eb7f69"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:52:40.169319 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.169275 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m24k\" (UniqueName: \"kubernetes.io/projected/e883afe2-3f49-48ab-9759-974238eb7f69-kube-api-access-5m24k\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:40.169319 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.169310 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e883afe2-3f49-48ab-9759-974238eb7f69-console-oauth-config\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:40.169319 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.169323 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e883afe2-3f49-48ab-9759-974238eb7f69-console-serving-cert\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:40.169558 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.169334 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-trusted-ca-bundle\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:40.169558 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.169348 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-console-config\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:40.169558 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.169361 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-service-ca\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:40.169558 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.169372 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e883afe2-3f49-48ab-9759-974238eb7f69-oauth-serving-cert\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:40.720924 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.720888 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbjqw" Jul 16 12:52:40.782117 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.782082 2565 generic.go:356] "Generic (PLEG): container finished" podID="e883afe2-3f49-48ab-9759-974238eb7f69" containerID="533ef659f6f67b02db1de90fbfe39920c958f4c609f96c84852937be39c62a7a" exitCode=0 Jul 16 12:52:40.782644 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.782138 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bb975cf7-qgz42" Jul 16 12:52:40.782644 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.782199 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bb975cf7-qgz42" event={"ID":"e883afe2-3f49-48ab-9759-974238eb7f69","Type":"ContainerDied","Data":"533ef659f6f67b02db1de90fbfe39920c958f4c609f96c84852937be39c62a7a"} Jul 16 12:52:40.782644 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.782292 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bb975cf7-qgz42" event={"ID":"e883afe2-3f49-48ab-9759-974238eb7f69","Type":"ContainerDied","Data":"f1ed133704a2c5c6ad8497d1b586a8bf8a02f06167bf856ad42e4724ba7b14fa"} Jul 16 12:52:40.782644 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.782320 2565 scope.go:117] "RemoveContainer" containerID="533ef659f6f67b02db1de90fbfe39920c958f4c609f96c84852937be39c62a7a" Jul 16 12:52:40.796058 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.796038 2565 scope.go:117] "RemoveContainer" containerID="533ef659f6f67b02db1de90fbfe39920c958f4c609f96c84852937be39c62a7a" Jul 16 12:52:40.796373 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:52:40.796351 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533ef659f6f67b02db1de90fbfe39920c958f4c609f96c84852937be39c62a7a\": container with ID starting with 533ef659f6f67b02db1de90fbfe39920c958f4c609f96c84852937be39c62a7a not found: ID does not exist" containerID="533ef659f6f67b02db1de90fbfe39920c958f4c609f96c84852937be39c62a7a" Jul 16 12:52:40.796483 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.796384 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533ef659f6f67b02db1de90fbfe39920c958f4c609f96c84852937be39c62a7a"} err="failed to get container status \"533ef659f6f67b02db1de90fbfe39920c958f4c609f96c84852937be39c62a7a\": rpc error: code = NotFound desc = could not find container \"533ef659f6f67b02db1de90fbfe39920c958f4c609f96c84852937be39c62a7a\": container with ID starting with 533ef659f6f67b02db1de90fbfe39920c958f4c609f96c84852937be39c62a7a not found: ID does not exist" Jul 16 12:52:40.804160 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.804134 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9bb975cf7-qgz42"] Jul 16 12:52:40.809008 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:40.808920 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9bb975cf7-qgz42"] Jul 16 12:52:42.547141 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:42.547107 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e883afe2-3f49-48ab-9759-974238eb7f69" path="/var/lib/kubelet/pods/e883afe2-3f49-48ab-9759-974238eb7f69/volumes" Jul 16 12:52:42.796319 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:42.796279 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w9d5v" event={"ID":"58dca71d-bb6d-406f-9699-fe45c5d9bbed","Type":"ContainerStarted","Data":"770533437ad4fc4ef73e1c7290fa806dab216db171ebd91049deea1808d33393"} Jul 16 12:52:42.796472 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:42.796342 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w9d5v" Jul 16 12:52:42.816376 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:42.816272 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w9d5v" podStartSLOduration=2.347808843 podStartE2EDuration="5.816256758s" totalImagesPullingTime="3.468447915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:52:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:52:42.814659894 +0000 UTC m=+470.809800668" watchObservedRunningTime="2026-07-16 12:52:42.816256758 +0000 UTC m=+470.811397514" Jul 16 12:52:44.744613 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:44.744585 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" Jul 16 12:52:53.808914 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:53.808885 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w9d5v" Jul 16 12:52:54.657183 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.657149 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg"] Jul 16 12:52:54.657462 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.657419 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" podUID="a53ed1d2-ef85-4ae3-aecd-c85c17661657" containerName="manager" containerID="cri-o://cb0209abb79c93f3acf3e5e2c8dd763b351cac82186878b025e309613b060ddd" gracePeriod=2 Jul 16 12:52:54.666465 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.666438 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg"] Jul 16 12:52:54.688410 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.688383 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l54v4"] Jul 16 12:52:54.688781 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.688766 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e883afe2-3f49-48ab-9759-974238eb7f69" containerName="console" Jul 16 12:52:54.688861 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.688785 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e883afe2-3f49-48ab-9759-974238eb7f69" containerName="console" Jul 16 12:52:54.688861 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.688799 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53ed1d2-ef85-4ae3-aecd-c85c17661657" containerName="manager" Jul 16 12:52:54.688861 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.688807 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53ed1d2-ef85-4ae3-aecd-c85c17661657" containerName="manager" Jul 16 12:52:54.689008 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.688888 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e883afe2-3f49-48ab-9759-974238eb7f69" containerName="console" Jul 16 12:52:54.689008 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.688901 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a53ed1d2-ef85-4ae3-aecd-c85c17661657" containerName="manager" Jul 16 12:52:54.692173 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.692149 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l54v4" Jul 16 12:52:54.694713 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:52:54.694687 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ltfbg\" is forbidden: User \"system:node:ip-10-0-138-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-198.ec2.internal' and this object" podUID="a53ed1d2-ef85-4ae3-aecd-c85c17661657" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" Jul 16 12:52:54.703923 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.703901 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l54v4"] Jul 16 12:52:54.798001 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.797970 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2dpl\" (UniqueName: \"kubernetes.io/projected/477a480a-9ea5-46db-9886-880006f6f9d0-kube-api-access-x2dpl\") pod \"limitador-operator-controller-manager-859d7b78bd-l54v4\" (UID: \"477a480a-9ea5-46db-9886-880006f6f9d0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l54v4" Jul 16 12:52:54.867472 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.867441 2565 generic.go:356] "Generic (PLEG): container finished" podID="a53ed1d2-ef85-4ae3-aecd-c85c17661657" containerID="cb0209abb79c93f3acf3e5e2c8dd763b351cac82186878b025e309613b060ddd" exitCode=0 Jul 16 12:52:54.898912 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.898876 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2dpl\" (UniqueName: \"kubernetes.io/projected/477a480a-9ea5-46db-9886-880006f6f9d0-kube-api-access-x2dpl\") pod \"limitador-operator-controller-manager-859d7b78bd-l54v4\" (UID: \"477a480a-9ea5-46db-9886-880006f6f9d0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l54v4" Jul 16 12:52:54.912355 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.912298 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2dpl\" (UniqueName: \"kubernetes.io/projected/477a480a-9ea5-46db-9886-880006f6f9d0-kube-api-access-x2dpl\") pod \"limitador-operator-controller-manager-859d7b78bd-l54v4\" (UID: \"477a480a-9ea5-46db-9886-880006f6f9d0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l54v4" Jul 16 12:52:54.926789 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.926766 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" Jul 16 12:52:54.928927 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:52:54.928906 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ltfbg\" is forbidden: User \"system:node:ip-10-0-138-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-198.ec2.internal' and this object" podUID="a53ed1d2-ef85-4ae3-aecd-c85c17661657" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" Jul 16 12:52:54.999521 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:54.999502 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8xz5\" (UniqueName: \"kubernetes.io/projected/a53ed1d2-ef85-4ae3-aecd-c85c17661657-kube-api-access-m8xz5\") pod \"a53ed1d2-ef85-4ae3-aecd-c85c17661657\" (UID: \"a53ed1d2-ef85-4ae3-aecd-c85c17661657\") " Jul 16 12:52:55.001525 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:55.001500 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53ed1d2-ef85-4ae3-aecd-c85c17661657-kube-api-access-m8xz5" (OuterVolumeSpecName: "kube-api-access-m8xz5") pod "a53ed1d2-ef85-4ae3-aecd-c85c17661657" (UID: "a53ed1d2-ef85-4ae3-aecd-c85c17661657"). InnerVolumeSpecName "kube-api-access-m8xz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:52:55.029637 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:55.029612 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l54v4" Jul 16 12:52:55.100965 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:55.100935 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8xz5\" (UniqueName: \"kubernetes.io/projected/a53ed1d2-ef85-4ae3-aecd-c85c17661657-kube-api-access-m8xz5\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:52:55.166571 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:55.166545 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l54v4"] Jul 16 12:52:55.168261 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:52:55.168198 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477a480a_9ea5_46db_9886_880006f6f9d0.slice/crio-678e39721e94d212def28d49c14f425c510bba6ec9fd14b309600fb08d2b09a9 WatchSource:0}: Error finding container 678e39721e94d212def28d49c14f425c510bba6ec9fd14b309600fb08d2b09a9: Status 404 returned error can't find the container with id 678e39721e94d212def28d49c14f425c510bba6ec9fd14b309600fb08d2b09a9 Jul 16 12:52:55.744163 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:55.744119 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" podUID="a53ed1d2-ef85-4ae3-aecd-c85c17661657" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.41:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 12:52:55.874819 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:55.874792 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" Jul 16 12:52:55.875267 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:55.874851 2565 scope.go:117] "RemoveContainer" containerID="cb0209abb79c93f3acf3e5e2c8dd763b351cac82186878b025e309613b060ddd" Jul 16 12:52:55.876600 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:55.876577 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l54v4" event={"ID":"477a480a-9ea5-46db-9886-880006f6f9d0","Type":"ContainerStarted","Data":"32ce85f3c30e4c642bb4030fb234c1624c97baf9fc04bc378f31c72a2d75f5f1"} Jul 16 12:52:55.876726 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:55.876618 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l54v4" event={"ID":"477a480a-9ea5-46db-9886-880006f6f9d0","Type":"ContainerStarted","Data":"678e39721e94d212def28d49c14f425c510bba6ec9fd14b309600fb08d2b09a9"} Jul 16 12:52:55.876795 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:55.876712 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l54v4" Jul 16 12:52:55.877360 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:52:55.877338 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ltfbg\" is forbidden: User \"system:node:ip-10-0-138-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-198.ec2.internal' and this object" podUID="a53ed1d2-ef85-4ae3-aecd-c85c17661657" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" Jul 16 12:52:55.904496 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:55.904449 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l54v4" podStartSLOduration=1.904432367 podStartE2EDuration="1.904432367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:52:55.902181597 +0000 UTC m=+483.897322359" watchObservedRunningTime="2026-07-16 12:52:55.904432367 +0000 UTC m=+483.899573123" Jul 16 12:52:55.905568 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:52:55.905545 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ltfbg\" is forbidden: User \"system:node:ip-10-0-138-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-198.ec2.internal' and this object" podUID="a53ed1d2-ef85-4ae3-aecd-c85c17661657" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" Jul 16 12:52:55.907462 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:52:55.907437 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ltfbg\" is forbidden: User \"system:node:ip-10-0-138-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-198.ec2.internal' and this object" podUID="a53ed1d2-ef85-4ae3-aecd-c85c17661657" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ltfbg" Jul 16 12:52:56.545041 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:52:56.545006 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a53ed1d2-ef85-4ae3-aecd-c85c17661657" path="/var/lib/kubelet/pods/a53ed1d2-ef85-4ae3-aecd-c85c17661657/volumes" Jul 16 12:53:06.885397 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:06.885366 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l54v4" Jul 16 12:53:41.782536 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.782501 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm"] Jul 16 12:53:41.789337 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.789319 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.796674 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.796652 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gd5n6\"" Jul 16 12:53:41.799221 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.799198 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.799340 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.799297 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.799379 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.799343 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nwcd\" (UniqueName: \"kubernetes.io/projected/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-kube-api-access-6nwcd\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.799379 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.799364 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.799474 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.799398 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.799474 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.799420 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.799534 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.799469 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.799534 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.799494 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.799534 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.799531 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.804412 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.804385 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm"] Jul 16 12:53:41.900852 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.900816 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.900852 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.900853 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.901090 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.900881 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.901090 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.901007 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.901090 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.901041 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.901090 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.901086 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nwcd\" (UniqueName: \"kubernetes.io/projected/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-kube-api-access-6nwcd\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.901342 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.901119 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.901342 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.901287 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.901342 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.901301 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.901342 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.901323 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.901553 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.901523 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.901685 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.901659 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.901836 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.901698 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.901836 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.901720 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.903363 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.903341 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.903875 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.903859 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.910702 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.910675 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:41.911545 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:41.911526 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nwcd\" (UniqueName: \"kubernetes.io/projected/0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb-kube-api-access-6nwcd\") pod \"maas-default-gateway-openshift-default-56654dcf7-nwcbm\" (UID: \"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:42.100070 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:42.099969 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:42.253993 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:42.253966 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm"] Jul 16 12:53:42.255873 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:53:42.255845 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd02b7f_5de8_46bb_b0ad_9dff6f0433eb.slice/crio-0c233ffe5a9b933512cd24dd3ae53c90572522bf15cfe791e921ab6fa7c3f752 WatchSource:0}: Error finding container 0c233ffe5a9b933512cd24dd3ae53c90572522bf15cfe791e921ab6fa7c3f752: Status 404 returned error can't find the container with id 0c233ffe5a9b933512cd24dd3ae53c90572522bf15cfe791e921ab6fa7c3f752 Jul 16 12:53:42.257946 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:42.257910 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 16 12:53:42.258033 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:42.257976 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 16 12:53:42.258033 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:42.258007 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 16 12:53:43.133766 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:43.133724 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" event={"ID":"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb","Type":"ContainerStarted","Data":"1b9e4fc581791bf5f49edc12310b2ab9cd3abbf07bedbd96a493911b99883280"} Jul 16 12:53:43.134150 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:43.133819 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" event={"ID":"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb","Type":"ContainerStarted","Data":"0c233ffe5a9b933512cd24dd3ae53c90572522bf15cfe791e921ab6fa7c3f752"} Jul 16 12:53:43.187013 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:43.186953 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" podStartSLOduration=2.186937904 podStartE2EDuration="2.186937904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:53:43.185975128 +0000 UTC m=+531.181115888" watchObservedRunningTime="2026-07-16 12:53:43.186937904 +0000 UTC m=+531.182078663" Jul 16 12:53:44.101053 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:44.100972 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:44.105945 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:44.105919 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:44.138792 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:44.138763 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:44.139746 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:44.139727 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:53:45.225211 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:45.225179 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-x8nlz"] Jul 16 12:53:45.228445 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:45.228429 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-x8nlz" Jul 16 12:53:45.245430 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:45.245405 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xpgq8\"" Jul 16 12:53:45.270028 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:45.269996 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-x8nlz"] Jul 16 12:53:45.331066 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:45.331025 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djfd6\" (UniqueName: \"kubernetes.io/projected/e296f59b-61ae-4189-aec4-046ce63cb74f-kube-api-access-djfd6\") pod \"authorino-58cd955675-x8nlz\" (UID: \"e296f59b-61ae-4189-aec4-046ce63cb74f\") " pod="kuadrant-system/authorino-58cd955675-x8nlz" Jul 16 12:53:45.432226 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:45.432189 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djfd6\" (UniqueName: \"kubernetes.io/projected/e296f59b-61ae-4189-aec4-046ce63cb74f-kube-api-access-djfd6\") pod \"authorino-58cd955675-x8nlz\" (UID: \"e296f59b-61ae-4189-aec4-046ce63cb74f\") " pod="kuadrant-system/authorino-58cd955675-x8nlz" Jul 16 12:53:45.447215 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:45.447185 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djfd6\" (UniqueName: \"kubernetes.io/projected/e296f59b-61ae-4189-aec4-046ce63cb74f-kube-api-access-djfd6\") pod \"authorino-58cd955675-x8nlz\" (UID: \"e296f59b-61ae-4189-aec4-046ce63cb74f\") " pod="kuadrant-system/authorino-58cd955675-x8nlz" Jul 16 12:53:45.538137 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:45.538105 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-x8nlz" Jul 16 12:53:45.677351 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:45.677321 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-x8nlz"] Jul 16 12:53:45.680257 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:53:45.680214 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode296f59b_61ae_4189_aec4_046ce63cb74f.slice/crio-04bc9f896d5b2e948aabb4ce4e35572b2e5a821f725b32a059a082bd85789693 WatchSource:0}: Error finding container 04bc9f896d5b2e948aabb4ce4e35572b2e5a821f725b32a059a082bd85789693: Status 404 returned error can't find the container with id 04bc9f896d5b2e948aabb4ce4e35572b2e5a821f725b32a059a082bd85789693 Jul 16 12:53:46.151039 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:46.151008 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-x8nlz" event={"ID":"e296f59b-61ae-4189-aec4-046ce63cb74f","Type":"ContainerStarted","Data":"04bc9f896d5b2e948aabb4ce4e35572b2e5a821f725b32a059a082bd85789693"} Jul 16 12:53:49.174157 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:49.174120 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-x8nlz" event={"ID":"e296f59b-61ae-4189-aec4-046ce63cb74f","Type":"ContainerStarted","Data":"4d46c3dbed4bb5787a7fbf1bf94e6f525ee67aaf35243ee819704ba8f7e85760"} Jul 16 12:53:49.200506 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:49.200448 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-x8nlz" podStartSLOduration=0.904356389 podStartE2EDuration="4.200430171s" totalImagesPullingTime="3.296073782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:53:49.198049845 +0000 UTC m=+537.193190605" watchObservedRunningTime="2026-07-16 12:53:49.200430171 +0000 UTC m=+537.195570930" Jul 16 12:53:50.343214 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:50.343174 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-x8nlz"] Jul 16 12:53:51.184614 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:51.184566 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-x8nlz" podUID="e296f59b-61ae-4189-aec4-046ce63cb74f" containerName="authorino" containerID="cri-o://4d46c3dbed4bb5787a7fbf1bf94e6f525ee67aaf35243ee819704ba8f7e85760" gracePeriod=30 Jul 16 12:53:51.440476 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:51.440399 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-x8nlz" Jul 16 12:53:51.489484 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:51.489445 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djfd6\" (UniqueName: \"kubernetes.io/projected/e296f59b-61ae-4189-aec4-046ce63cb74f-kube-api-access-djfd6\") pod \"e296f59b-61ae-4189-aec4-046ce63cb74f\" (UID: \"e296f59b-61ae-4189-aec4-046ce63cb74f\") " Jul 16 12:53:51.491668 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:51.491638 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e296f59b-61ae-4189-aec4-046ce63cb74f-kube-api-access-djfd6" (OuterVolumeSpecName: "kube-api-access-djfd6") pod "e296f59b-61ae-4189-aec4-046ce63cb74f" (UID: "e296f59b-61ae-4189-aec4-046ce63cb74f"). InnerVolumeSpecName "kube-api-access-djfd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:53:51.591014 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:51.590982 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djfd6\" (UniqueName: \"kubernetes.io/projected/e296f59b-61ae-4189-aec4-046ce63cb74f-kube-api-access-djfd6\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 12:53:52.191118 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:52.191080 2565 generic.go:356] "Generic (PLEG): container finished" podID="e296f59b-61ae-4189-aec4-046ce63cb74f" containerID="4d46c3dbed4bb5787a7fbf1bf94e6f525ee67aaf35243ee819704ba8f7e85760" exitCode=0 Jul 16 12:53:52.191308 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:52.191137 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-x8nlz" Jul 16 12:53:52.191308 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:52.191178 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-x8nlz" event={"ID":"e296f59b-61ae-4189-aec4-046ce63cb74f","Type":"ContainerDied","Data":"4d46c3dbed4bb5787a7fbf1bf94e6f525ee67aaf35243ee819704ba8f7e85760"} Jul 16 12:53:52.191308 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:52.191253 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-x8nlz" event={"ID":"e296f59b-61ae-4189-aec4-046ce63cb74f","Type":"ContainerDied","Data":"04bc9f896d5b2e948aabb4ce4e35572b2e5a821f725b32a059a082bd85789693"} Jul 16 12:53:52.191308 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:52.191279 2565 scope.go:117] "RemoveContainer" containerID="4d46c3dbed4bb5787a7fbf1bf94e6f525ee67aaf35243ee819704ba8f7e85760" Jul 16 12:53:52.200656 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:52.200638 2565 scope.go:117] "RemoveContainer" containerID="4d46c3dbed4bb5787a7fbf1bf94e6f525ee67aaf35243ee819704ba8f7e85760" Jul 16 12:53:52.200888 ip-10-0-138-198 kubenswrapper[2565]: E0716 12:53:52.200873 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d46c3dbed4bb5787a7fbf1bf94e6f525ee67aaf35243ee819704ba8f7e85760\": container with ID starting with 4d46c3dbed4bb5787a7fbf1bf94e6f525ee67aaf35243ee819704ba8f7e85760 not found: ID does not exist" containerID="4d46c3dbed4bb5787a7fbf1bf94e6f525ee67aaf35243ee819704ba8f7e85760" Jul 16 12:53:52.200944 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:52.200894 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d46c3dbed4bb5787a7fbf1bf94e6f525ee67aaf35243ee819704ba8f7e85760"} err="failed to get container status \"4d46c3dbed4bb5787a7fbf1bf94e6f525ee67aaf35243ee819704ba8f7e85760\": rpc error: code = NotFound desc = could not find container \"4d46c3dbed4bb5787a7fbf1bf94e6f525ee67aaf35243ee819704ba8f7e85760\": container with ID starting with 4d46c3dbed4bb5787a7fbf1bf94e6f525ee67aaf35243ee819704ba8f7e85760 not found: ID does not exist" Jul 16 12:53:52.216524 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:52.216503 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-x8nlz"] Jul 16 12:53:52.224243 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:52.224212 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-x8nlz"] Jul 16 12:53:52.549976 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:53:52.549944 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e296f59b-61ae-4189-aec4-046ce63cb74f" path="/var/lib/kubelet/pods/e296f59b-61ae-4189-aec4-046ce63cb74f/volumes" Jul 16 12:54:52.474714 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:54:52.474682 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 12:54:52.475276 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:54:52.474881 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 12:55:13.624544 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:13.624509 2565 generic.go:356] "Generic (PLEG): container finished" podID="0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb" containerID="1b9e4fc581791bf5f49edc12310b2ab9cd3abbf07bedbd96a493911b99883280" exitCode=0 Jul 16 12:55:13.624951 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:13.624583 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" event={"ID":"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb","Type":"ContainerDied","Data":"1b9e4fc581791bf5f49edc12310b2ab9cd3abbf07bedbd96a493911b99883280"} Jul 16 12:55:13.625009 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:13.624973 2565 scope.go:117] "RemoveContainer" containerID="1b9e4fc581791bf5f49edc12310b2ab9cd3abbf07bedbd96a493911b99883280" Jul 16 12:55:13.625978 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:13.625935 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 16 12:55:13.626088 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:13.626017 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 16 12:55:13.626088 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:13.626063 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 16 12:55:14.139342 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:14.139302 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:55:14.652258 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:14.646587 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" event={"ID":"0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb","Type":"ContainerStarted","Data":"4c82cd37debbc216fdb14b737178265231ac1f43ae854c7afed089a11f593317"} Jul 16 12:55:15.100899 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:15.100856 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:55:15.102088 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:15.102058 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-nwcbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 16 12:55:15.102256 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:15.102113 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" podUID="0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 16 12:55:16.101076 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:16.101043 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-nwcbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 16 12:55:16.101481 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:16.101096 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" podUID="0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 16 12:55:17.100680 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:17.100642 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-nwcbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 16 12:55:17.100862 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:17.100701 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" podUID="0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 16 12:55:18.101122 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:18.101089 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-nwcbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 16 12:55:18.101540 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:18.101145 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" podUID="0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 16 12:55:19.104895 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:19.104866 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:55:19.105373 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:19.105140 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:55:19.105933 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:19.105914 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nwcbm" Jul 16 12:55:35.613721 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.613691 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2"] Jul 16 12:55:35.614143 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.614041 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e296f59b-61ae-4189-aec4-046ce63cb74f" containerName="authorino" Jul 16 12:55:35.614143 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.614051 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e296f59b-61ae-4189-aec4-046ce63cb74f" containerName="authorino" Jul 16 12:55:35.614143 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.614115 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e296f59b-61ae-4189-aec4-046ce63cb74f" containerName="authorino" Jul 16 12:55:35.619309 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.618061 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.621486 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.621464 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 16 12:55:35.621746 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.621729 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 12:55:35.622304 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.622286 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 12:55:35.622403 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.622367 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nnbbp\"" Jul 16 12:55:35.630720 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.630696 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2"] Jul 16 12:55:35.748664 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.748604 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c7fb18-47a7-4e95-a8a7-74607faacae8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.748864 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.748700 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c7fb18-47a7-4e95-a8a7-74607faacae8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.748942 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.748863 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c7fb18-47a7-4e95-a8a7-74607faacae8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.749001 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.748977 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c7fb18-47a7-4e95-a8a7-74607faacae8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.749056 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.749034 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c7fb18-47a7-4e95-a8a7-74607faacae8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.749119 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.749066 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq99w\" (UniqueName: \"kubernetes.io/projected/84c7fb18-47a7-4e95-a8a7-74607faacae8-kube-api-access-nq99w\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.850210 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.850179 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c7fb18-47a7-4e95-a8a7-74607faacae8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.850404 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.850254 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c7fb18-47a7-4e95-a8a7-74607faacae8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.850404 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.850320 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c7fb18-47a7-4e95-a8a7-74607faacae8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.850404 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.850353 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c7fb18-47a7-4e95-a8a7-74607faacae8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.850404 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.850378 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq99w\" (UniqueName: \"kubernetes.io/projected/84c7fb18-47a7-4e95-a8a7-74607faacae8-kube-api-access-nq99w\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.850609 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.850419 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c7fb18-47a7-4e95-a8a7-74607faacae8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.850666 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.850650 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c7fb18-47a7-4e95-a8a7-74607faacae8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.850715 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.850670 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c7fb18-47a7-4e95-a8a7-74607faacae8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.850781 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.850735 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c7fb18-47a7-4e95-a8a7-74607faacae8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.852791 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.852773 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c7fb18-47a7-4e95-a8a7-74607faacae8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.852976 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.852956 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c7fb18-47a7-4e95-a8a7-74607faacae8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.860449 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.860431 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq99w\" (UniqueName: \"kubernetes.io/projected/84c7fb18-47a7-4e95-a8a7-74607faacae8-kube-api-access-nq99w\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pmpq2\" (UID: \"84c7fb18-47a7-4e95-a8a7-74607faacae8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:35.930769 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:35.930708 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:36.079867 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:36.079846 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2"] Jul 16 12:55:36.081466 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:55:36.081437 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c7fb18_47a7_4e95_a8a7_74607faacae8.slice/crio-5dc5a81a8ce8415d37df98c8e7ba9feace799925bfb11d37c958af191d6e179c WatchSource:0}: Error finding container 5dc5a81a8ce8415d37df98c8e7ba9feace799925bfb11d37c958af191d6e179c: Status 404 returned error can't find the container with id 5dc5a81a8ce8415d37df98c8e7ba9feace799925bfb11d37c958af191d6e179c Jul 16 12:55:36.082808 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:36.082786 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:55:36.766732 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:36.766689 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" event={"ID":"84c7fb18-47a7-4e95-a8a7-74607faacae8","Type":"ContainerStarted","Data":"5dc5a81a8ce8415d37df98c8e7ba9feace799925bfb11d37c958af191d6e179c"} Jul 16 12:55:37.773367 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:37.773326 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" event={"ID":"84c7fb18-47a7-4e95-a8a7-74607faacae8","Type":"ContainerStarted","Data":"51a70de22b4526c857cae1f769816c3142672e7cab85496ed58117fe436b0778"} Jul 16 12:55:37.803712 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:37.803655 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" podStartSLOduration=1.577959823 podStartE2EDuration="2.803638421s" totalImagesPullingTime="1.225678598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:55:37.80142653 +0000 UTC m=+645.796567313" watchObservedRunningTime="2026-07-16 12:55:37.803638421 +0000 UTC m=+645.798779181" Jul 16 12:55:45.196443 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.196408 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78"] Jul 16 12:55:45.203383 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.200825 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.203993 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.203967 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 16 12:55:45.210858 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.210835 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78"] Jul 16 12:55:45.337635 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.337600 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ddee62-e0ae-406f-aff9-b9089de0123e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.337807 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.337645 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ddee62-e0ae-406f-aff9-b9089de0123e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.337807 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.337736 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ddee62-e0ae-406f-aff9-b9089de0123e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.337807 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.337762 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ddee62-e0ae-406f-aff9-b9089de0123e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.337807 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.337787 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c86jt\" (UniqueName: \"kubernetes.io/projected/09ddee62-e0ae-406f-aff9-b9089de0123e-kube-api-access-c86jt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.337966 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.337809 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ddee62-e0ae-406f-aff9-b9089de0123e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.438552 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.438505 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ddee62-e0ae-406f-aff9-b9089de0123e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.438743 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.438566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ddee62-e0ae-406f-aff9-b9089de0123e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.438743 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.438629 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ddee62-e0ae-406f-aff9-b9089de0123e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.438743 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.438657 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ddee62-e0ae-406f-aff9-b9089de0123e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.438743 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.438680 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c86jt\" (UniqueName: \"kubernetes.io/projected/09ddee62-e0ae-406f-aff9-b9089de0123e-kube-api-access-c86jt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.438743 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.438708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ddee62-e0ae-406f-aff9-b9089de0123e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.439017 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.438963 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ddee62-e0ae-406f-aff9-b9089de0123e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.439094 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.439071 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ddee62-e0ae-406f-aff9-b9089de0123e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.439145 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.439112 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ddee62-e0ae-406f-aff9-b9089de0123e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.440908 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.440883 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ddee62-e0ae-406f-aff9-b9089de0123e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.441319 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.441300 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ddee62-e0ae-406f-aff9-b9089de0123e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.450631 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.450579 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c86jt\" (UniqueName: \"kubernetes.io/projected/09ddee62-e0ae-406f-aff9-b9089de0123e-kube-api-access-c86jt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-w2t78\" (UID: \"09ddee62-e0ae-406f-aff9-b9089de0123e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.514754 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.514730 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:45.655809 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.655781 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78"] Jul 16 12:55:45.657260 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:55:45.657218 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ddee62_e0ae_406f_aff9_b9089de0123e.slice/crio-47f90161232fbad88e32ef3ae70fe40669cceb4dd4d423daba2c0e9488d16ac5 WatchSource:0}: Error finding container 47f90161232fbad88e32ef3ae70fe40669cceb4dd4d423daba2c0e9488d16ac5: Status 404 returned error can't find the container with id 47f90161232fbad88e32ef3ae70fe40669cceb4dd4d423daba2c0e9488d16ac5 Jul 16 12:55:45.816523 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.816489 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" event={"ID":"09ddee62-e0ae-406f-aff9-b9089de0123e","Type":"ContainerStarted","Data":"47f90161232fbad88e32ef3ae70fe40669cceb4dd4d423daba2c0e9488d16ac5"} Jul 16 12:55:45.930957 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.930919 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:45.931143 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.930995 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:45.943473 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:45.943445 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:46.824100 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:46.824067 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" event={"ID":"09ddee62-e0ae-406f-aff9-b9089de0123e","Type":"ContainerStarted","Data":"6348ad0c6a1f5b6866e86f94d795ce1d508989a13531a386927efc5191f032db"} Jul 16 12:55:46.835515 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:46.835495 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pmpq2" Jul 16 12:55:46.877154 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:46.877103 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" podStartSLOduration=1.417251366 podStartE2EDuration="1.877084765s" totalImagesPullingTime="459.833399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:55:46.874354207 +0000 UTC m=+654.869494967" watchObservedRunningTime="2026-07-16 12:55:46.877084765 +0000 UTC m=+654.872225524" Jul 16 12:55:53.798194 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.798163 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8"] Jul 16 12:55:53.804492 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.804474 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.807100 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.807081 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 16 12:55:53.811889 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.811565 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8"] Jul 16 12:55:53.811889 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.811675 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2wxv\" (UniqueName: \"kubernetes.io/projected/356e28f2-b8fc-4634-bc08-27edf9f46bfe-kube-api-access-d2wxv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.811889 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.811717 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356e28f2-b8fc-4634-bc08-27edf9f46bfe-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.811889 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.811795 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/356e28f2-b8fc-4634-bc08-27edf9f46bfe-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.811889 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.811839 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/356e28f2-b8fc-4634-bc08-27edf9f46bfe-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.811889 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.811879 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/356e28f2-b8fc-4634-bc08-27edf9f46bfe-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.812206 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.811904 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/356e28f2-b8fc-4634-bc08-27edf9f46bfe-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.912802 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.912771 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2wxv\" (UniqueName: \"kubernetes.io/projected/356e28f2-b8fc-4634-bc08-27edf9f46bfe-kube-api-access-d2wxv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.912802 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.912805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356e28f2-b8fc-4634-bc08-27edf9f46bfe-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.913019 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.912837 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/356e28f2-b8fc-4634-bc08-27edf9f46bfe-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.913019 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.912877 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/356e28f2-b8fc-4634-bc08-27edf9f46bfe-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.913019 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.912916 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/356e28f2-b8fc-4634-bc08-27edf9f46bfe-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.913019 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.912941 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/356e28f2-b8fc-4634-bc08-27edf9f46bfe-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.913293 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.913269 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356e28f2-b8fc-4634-bc08-27edf9f46bfe-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.913420 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.913277 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/356e28f2-b8fc-4634-bc08-27edf9f46bfe-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.913420 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.913341 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/356e28f2-b8fc-4634-bc08-27edf9f46bfe-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.915365 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.915343 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/356e28f2-b8fc-4634-bc08-27edf9f46bfe-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.915561 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.915542 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/356e28f2-b8fc-4634-bc08-27edf9f46bfe-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:53.922695 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:53.922668 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2wxv\" (UniqueName: \"kubernetes.io/projected/356e28f2-b8fc-4634-bc08-27edf9f46bfe-kube-api-access-d2wxv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8\" (UID: \"356e28f2-b8fc-4634-bc08-27edf9f46bfe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:54.116440 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:54.116355 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:55:54.256561 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:54.256534 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8"] Jul 16 12:55:54.257926 ip-10-0-138-198 kubenswrapper[2565]: W0716 12:55:54.257897 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356e28f2_b8fc_4634_bc08_27edf9f46bfe.slice/crio-7da46d3c23273ae1e90498d9a8aaebdc10dc4c790849ee429fbce00ba9ea8dcb WatchSource:0}: Error finding container 7da46d3c23273ae1e90498d9a8aaebdc10dc4c790849ee429fbce00ba9ea8dcb: Status 404 returned error can't find the container with id 7da46d3c23273ae1e90498d9a8aaebdc10dc4c790849ee429fbce00ba9ea8dcb Jul 16 12:55:54.867676 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:54.867579 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" event={"ID":"356e28f2-b8fc-4634-bc08-27edf9f46bfe","Type":"ContainerStarted","Data":"d2b126cabacdec27ce8cb1cd1b0df67a7c266f18491d64108d520b5ab5d32ad2"} Jul 16 12:55:54.867676 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:54.867643 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" event={"ID":"356e28f2-b8fc-4634-bc08-27edf9f46bfe","Type":"ContainerStarted","Data":"7da46d3c23273ae1e90498d9a8aaebdc10dc4c790849ee429fbce00ba9ea8dcb"} Jul 16 12:55:54.893091 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:54.893046 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" podStartSLOduration=1.6175352859999999 podStartE2EDuration="1.893030621s" totalImagesPullingTime="275.495335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:55:54.890765466 +0000 UTC m=+662.885906240" watchObservedRunningTime="2026-07-16 12:55:54.893030621 +0000 UTC m=+662.888171382" Jul 16 12:55:55.515710 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:55.515670 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:55.515894 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:55.515730 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:55.528210 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:55.528190 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:55:55.884958 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:55:55.884873 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-w2t78" Jul 16 12:56:04.116863 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:56:04.116830 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:56:04.117279 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:56:04.116888 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:56:04.128901 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:56:04.128875 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:56:04.932676 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:56:04.932646 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8" Jul 16 12:59:52.510045 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:59:52.509965 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 12:59:52.511923 ip-10-0-138-198 kubenswrapper[2565]: I0716 12:59:52.511899 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 13:04:52.553703 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:04:52.553669 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 13:04:52.555621 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:04:52.555603 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 13:09:52.593109 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:09:52.593082 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 13:09:52.595622 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:09:52.594913 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 13:14:52.626976 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:14:52.626851 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 13:14:52.634701 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:14:52.630500 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 13:19:52.661166 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:19:52.661053 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 13:19:52.665471 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:19:52.665367 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 13:20:46.687297 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.687206 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9"] Jul 16 13:20:46.690733 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.690715 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.693365 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.693348 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a1f86fcf\"/\"openshift-service-ca.crt\"" Jul 16 13:20:46.693514 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.693496 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a1f86fcf\"/\"default-dockercfg-jrcwc\"" Jul 16 13:20:46.694332 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.694300 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a1f86fcf\"/\"kube-root-ca.crt\"" Jul 16 13:20:46.694433 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.694406 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a1f86fcf\"/\"e2e-lifecycle-model-a1f86fcf-kserve-self-signed-certs\"" Jul 16 13:20:46.700803 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.700774 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9"] Jul 16 13:20:46.772971 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.772934 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-home\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.772971 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.772974 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-tmp-dir\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.773189 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.772995 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfkdk\" (UniqueName: \"kubernetes.io/projected/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-kube-api-access-jfkdk\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.773189 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.773067 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-dshm\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.773189 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.773097 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-model-cache\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.773189 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.773116 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-tls-certs\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.874016 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.873979 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-dshm\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.874016 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.874018 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-model-cache\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.874386 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.874044 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-tls-certs\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.874386 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.874137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-home\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.874386 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.874175 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-tmp-dir\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.874386 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.874202 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkdk\" (UniqueName: \"kubernetes.io/projected/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-kube-api-access-jfkdk\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.874592 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.874454 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-model-cache\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.874592 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.874531 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-home\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.874592 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.874574 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-tmp-dir\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.876457 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.876436 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-dshm\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.876597 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.876582 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-tls-certs\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:46.883112 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:46.883092 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfkdk\" (UniqueName: \"kubernetes.io/projected/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-kube-api-access-jfkdk\") pod \"e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:47.001210 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:47.001099 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:47.146577 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:47.146550 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9"] Jul 16 13:20:47.148677 ip-10-0-138-198 kubenswrapper[2565]: W0716 13:20:47.148649 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e3fb40f_457d_4a3e_8fe9_018fb20669d8.slice/crio-3b65abdc495fc040ae22ba5fcf0384b5b5458525b1b9cba8ea7ddf8f5cbc5af0 WatchSource:0}: Error finding container 3b65abdc495fc040ae22ba5fcf0384b5b5458525b1b9cba8ea7ddf8f5cbc5af0: Status 404 returned error can't find the container with id 3b65abdc495fc040ae22ba5fcf0384b5b5458525b1b9cba8ea7ddf8f5cbc5af0 Jul 16 13:20:48.036778 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:48.036741 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" event={"ID":"0e3fb40f-457d-4a3e-8fe9-018fb20669d8","Type":"ContainerStarted","Data":"515c28ed612e00e0cd035a0132238172289e3709224772ec610be4330d46104a"} Jul 16 13:20:48.037193 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:48.036791 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" event={"ID":"0e3fb40f-457d-4a3e-8fe9-018fb20669d8","Type":"ContainerStarted","Data":"3b65abdc495fc040ae22ba5fcf0384b5b5458525b1b9cba8ea7ddf8f5cbc5af0"} Jul 16 13:20:48.057010 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:48.056963 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" podStartSLOduration=2.056948784 podStartE2EDuration="2.056948784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:20:48.05416401 +0000 UTC m=+2156.049304769" watchObservedRunningTime="2026-07-16 13:20:48.056948784 +0000 UTC m=+2156.052089544" Jul 16 13:20:57.001755 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:57.001711 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:57.002292 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:57.001776 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:57.014455 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:57.014432 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:20:57.103019 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:20:57.102990 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:21:44.072589 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:44.072554 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9"] Jul 16 13:21:59.094537 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.094478 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" podUID="0e3fb40f-457d-4a3e-8fe9-018fb20669d8" containerName="main" containerID="cri-o://515c28ed612e00e0cd035a0132238172289e3709224772ec610be4330d46104a" gracePeriod=45 Jul 16 13:21:59.353041 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.352985 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:21:59.434189 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.434160 2565 generic.go:356] "Generic (PLEG): container finished" podID="0e3fb40f-457d-4a3e-8fe9-018fb20669d8" containerID="515c28ed612e00e0cd035a0132238172289e3709224772ec610be4330d46104a" exitCode=0 Jul 16 13:21:59.434355 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.434200 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" Jul 16 13:21:59.434355 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.434213 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" event={"ID":"0e3fb40f-457d-4a3e-8fe9-018fb20669d8","Type":"ContainerDied","Data":"515c28ed612e00e0cd035a0132238172289e3709224772ec610be4330d46104a"} Jul 16 13:21:59.434355 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.434263 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9" event={"ID":"0e3fb40f-457d-4a3e-8fe9-018fb20669d8","Type":"ContainerDied","Data":"3b65abdc495fc040ae22ba5fcf0384b5b5458525b1b9cba8ea7ddf8f5cbc5af0"} Jul 16 13:21:59.434355 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.434281 2565 scope.go:117] "RemoveContainer" containerID="515c28ed612e00e0cd035a0132238172289e3709224772ec610be4330d46104a" Jul 16 13:21:59.440531 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.440508 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-tls-certs\") pod \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " Jul 16 13:21:59.440664 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.440591 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-model-cache\") pod \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " Jul 16 13:21:59.440664 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.440624 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfkdk\" (UniqueName: \"kubernetes.io/projected/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-kube-api-access-jfkdk\") pod \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " Jul 16 13:21:59.440664 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.440648 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-dshm\") pod \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " Jul 16 13:21:59.440838 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.440726 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-home\") pod \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " Jul 16 13:21:59.440838 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.440769 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-tmp-dir\") pod \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\" (UID: \"0e3fb40f-457d-4a3e-8fe9-018fb20669d8\") " Jul 16 13:21:59.440955 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.440899 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e3fb40f-457d-4a3e-8fe9-018fb20669d8" (UID: "0e3fb40f-457d-4a3e-8fe9-018fb20669d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:21:59.441047 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.441013 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-home" (OuterVolumeSpecName: "home") pod "0e3fb40f-457d-4a3e-8fe9-018fb20669d8" (UID: "0e3fb40f-457d-4a3e-8fe9-018fb20669d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:21:59.441113 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.441073 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-model-cache\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:21:59.441355 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.441112 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e3fb40f-457d-4a3e-8fe9-018fb20669d8" (UID: "0e3fb40f-457d-4a3e-8fe9-018fb20669d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:21:59.442894 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.442868 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e3fb40f-457d-4a3e-8fe9-018fb20669d8" (UID: "0e3fb40f-457d-4a3e-8fe9-018fb20669d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:21:59.442986 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.442899 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-kube-api-access-jfkdk" (OuterVolumeSpecName: "kube-api-access-jfkdk") pod "0e3fb40f-457d-4a3e-8fe9-018fb20669d8" (UID: "0e3fb40f-457d-4a3e-8fe9-018fb20669d8"). InnerVolumeSpecName "kube-api-access-jfkdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:21:59.443085 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.443069 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-dshm" (OuterVolumeSpecName: "dshm") pod "0e3fb40f-457d-4a3e-8fe9-018fb20669d8" (UID: "0e3fb40f-457d-4a3e-8fe9-018fb20669d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:21:59.444310 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.444286 2565 scope.go:117] "RemoveContainer" containerID="515c28ed612e00e0cd035a0132238172289e3709224772ec610be4330d46104a" Jul 16 13:21:59.444625 ip-10-0-138-198 kubenswrapper[2565]: E0716 13:21:59.444607 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515c28ed612e00e0cd035a0132238172289e3709224772ec610be4330d46104a\": container with ID starting with 515c28ed612e00e0cd035a0132238172289e3709224772ec610be4330d46104a not found: ID does not exist" containerID="515c28ed612e00e0cd035a0132238172289e3709224772ec610be4330d46104a" Jul 16 13:21:59.444709 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.444633 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515c28ed612e00e0cd035a0132238172289e3709224772ec610be4330d46104a"} err="failed to get container status \"515c28ed612e00e0cd035a0132238172289e3709224772ec610be4330d46104a\": rpc error: code = NotFound desc = could not find container \"515c28ed612e00e0cd035a0132238172289e3709224772ec610be4330d46104a\": container with ID starting with 515c28ed612e00e0cd035a0132238172289e3709224772ec610be4330d46104a not found: ID does not exist" Jul 16 13:21:59.541572 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.541532 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-home\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:21:59.541572 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.541565 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-tmp-dir\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:21:59.541572 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.541575 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-tls-certs\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:21:59.541794 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.541583 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfkdk\" (UniqueName: \"kubernetes.io/projected/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-kube-api-access-jfkdk\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:21:59.541794 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.541591 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3fb40f-457d-4a3e-8fe9-018fb20669d8-dshm\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:21:59.759991 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.759962 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9"] Jul 16 13:21:59.764910 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:21:59.764888 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-a1f86fcf/e2e-lifecycle-model-a1f86fcf-kserve-9b688cb67-z6ng9"] Jul 16 13:22:00.545578 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:22:00.545542 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e3fb40f-457d-4a3e-8fe9-018fb20669d8" path="/var/lib/kubelet/pods/0e3fb40f-457d-4a3e-8fe9-018fb20669d8/volumes" Jul 16 13:24:45.217296 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.217256 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4"] Jul 16 13:24:45.217784 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.217635 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e3fb40f-457d-4a3e-8fe9-018fb20669d8" containerName="main" Jul 16 13:24:45.217784 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.217647 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e3fb40f-457d-4a3e-8fe9-018fb20669d8" containerName="main" Jul 16 13:24:45.217784 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.217711 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e3fb40f-457d-4a3e-8fe9-018fb20669d8" containerName="main" Jul 16 13:24:45.220827 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.220812 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.224320 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.224299 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ab35b1\"/\"default-dockercfg-s2gfw\"" Jul 16 13:24:45.224418 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.224299 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ab35b1\"/\"kube-root-ca.crt\"" Jul 16 13:24:45.224496 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.224480 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ab35b1\"/\"test-model-ab35b1-kserve-self-signed-certs\"" Jul 16 13:24:45.224556 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.224489 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ab35b1\"/\"openshift-service-ca.crt\"" Jul 16 13:24:45.237216 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.237197 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4"] Jul 16 13:24:45.335979 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.335946 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-home\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.335979 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.335978 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-model-cache\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.336202 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.336009 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-tmp-dir\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.336202 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.336101 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-dshm\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.336202 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.336133 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbkbl\" (UniqueName: \"kubernetes.io/projected/8f1f0152-be54-4b47-ae12-cbc80dafceb3-kube-api-access-xbkbl\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.336202 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.336183 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f1f0152-be54-4b47-ae12-cbc80dafceb3-tls-certs\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.437411 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.437371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f1f0152-be54-4b47-ae12-cbc80dafceb3-tls-certs\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.437628 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.437464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-home\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.437628 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.437488 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-model-cache\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.437628 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.437540 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-tmp-dir\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.437628 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.437612 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-dshm\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.437853 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.437643 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbkbl\" (UniqueName: \"kubernetes.io/projected/8f1f0152-be54-4b47-ae12-cbc80dafceb3-kube-api-access-xbkbl\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.438025 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.437993 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-home\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.438222 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.438201 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-model-cache\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.438487 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.438465 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-tmp-dir\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.440222 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.440191 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f1f0152-be54-4b47-ae12-cbc80dafceb3-tls-certs\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.440350 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.440283 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-dshm\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.448151 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.448127 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbkbl\" (UniqueName: \"kubernetes.io/projected/8f1f0152-be54-4b47-ae12-cbc80dafceb3-kube-api-access-xbkbl\") pod \"test-model-ab35b1-kserve-fb94dc7c-gkvq4\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.530431 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.530406 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:45.675647 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:45.675620 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4"] Jul 16 13:24:45.677123 ip-10-0-138-198 kubenswrapper[2565]: W0716 13:24:45.677097 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f1f0152_be54_4b47_ae12_cbc80dafceb3.slice/crio-0bc90b4b36c49e2aac9aba534693d337b348f0764fd624ee5e2878cf0c53482c WatchSource:0}: Error finding container 0bc90b4b36c49e2aac9aba534693d337b348f0764fd624ee5e2878cf0c53482c: Status 404 returned error can't find the container with id 0bc90b4b36c49e2aac9aba534693d337b348f0764fd624ee5e2878cf0c53482c Jul 16 13:24:46.355888 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:46.355851 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" event={"ID":"8f1f0152-be54-4b47-ae12-cbc80dafceb3","Type":"ContainerStarted","Data":"3a2078873ccd51603c82e8f84738c18ec069a6475556a1c363cc5a982a89f36b"} Jul 16 13:24:46.356304 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:46.355895 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" event={"ID":"8f1f0152-be54-4b47-ae12-cbc80dafceb3","Type":"ContainerStarted","Data":"0bc90b4b36c49e2aac9aba534693d337b348f0764fd624ee5e2878cf0c53482c"} Jul 16 13:24:46.381663 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:46.381615 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" podStartSLOduration=1.381599244 podStartE2EDuration="1.381599244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:24:46.379074637 +0000 UTC m=+2394.374215407" watchObservedRunningTime="2026-07-16 13:24:46.381599244 +0000 UTC m=+2394.376740019" Jul 16 13:24:52.695437 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:52.695330 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 13:24:52.710185 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:52.702290 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 13:24:55.531122 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:55.531086 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:55.531535 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:55.531135 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:55.543954 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:55.543930 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:24:56.421472 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:24:56.421444 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:25:09.468544 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:09.468513 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4"] Jul 16 13:25:09.606353 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:09.606317 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9"] Jul 16 13:25:09.609893 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:09.609876 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9" Jul 16 13:25:09.612014 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:09.611993 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9kff4\"" Jul 16 13:25:09.612147 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:09.612111 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 13:25:09.618171 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:09.618146 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9"] Jul 16 13:25:09.662108 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:09.662070 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1c2b14-07f4-4150-8bbf-6114e3498ebc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9\" (UID: \"5e1c2b14-07f4-4150-8bbf-6114e3498ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9" Jul 16 13:25:09.763178 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:09.763099 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1c2b14-07f4-4150-8bbf-6114e3498ebc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9\" (UID: \"5e1c2b14-07f4-4150-8bbf-6114e3498ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9" Jul 16 13:25:09.763712 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:09.763694 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1c2b14-07f4-4150-8bbf-6114e3498ebc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9\" (UID: \"5e1c2b14-07f4-4150-8bbf-6114e3498ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9" Jul 16 13:25:09.920885 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:09.920854 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9" Jul 16 13:25:10.068243 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:10.068210 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9"] Jul 16 13:25:10.070071 ip-10-0-138-198 kubenswrapper[2565]: W0716 13:25:10.070030 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1c2b14_07f4_4150_8bbf_6114e3498ebc.slice/crio-a6e91c99fa213c9b48692d9581b67828385fbf9429bb95d9091cb3cc632ac5c5 WatchSource:0}: Error finding container a6e91c99fa213c9b48692d9581b67828385fbf9429bb95d9091cb3cc632ac5c5: Status 404 returned error can't find the container with id a6e91c99fa213c9b48692d9581b67828385fbf9429bb95d9091cb3cc632ac5c5 Jul 16 13:25:10.072034 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:10.072016 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 13:25:10.495966 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:10.495876 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9" event={"ID":"5e1c2b14-07f4-4150-8bbf-6114e3498ebc","Type":"ContainerStarted","Data":"a6e91c99fa213c9b48692d9581b67828385fbf9429bb95d9091cb3cc632ac5c5"} Jul 16 13:25:13.513923 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:13.513836 2565 generic.go:356] "Generic (PLEG): container finished" podID="5e1c2b14-07f4-4150-8bbf-6114e3498ebc" containerID="7a98ed4311f26368913c0a974eb27532017bd2a05b9262fbf5042a41fce2f4a2" exitCode=0 Jul 16 13:25:13.514296 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:13.513919 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9" event={"ID":"5e1c2b14-07f4-4150-8bbf-6114e3498ebc","Type":"ContainerDied","Data":"7a98ed4311f26368913c0a974eb27532017bd2a05b9262fbf5042a41fce2f4a2"} Jul 16 13:25:14.661693 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:14.661670 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9" Jul 16 13:25:14.710023 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:14.709994 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1c2b14-07f4-4150-8bbf-6114e3498ebc-maas-api-service-ca\") pod \"5e1c2b14-07f4-4150-8bbf-6114e3498ebc\" (UID: \"5e1c2b14-07f4-4150-8bbf-6114e3498ebc\") " Jul 16 13:25:14.710361 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:14.710339 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e1c2b14-07f4-4150-8bbf-6114e3498ebc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e1c2b14-07f4-4150-8bbf-6114e3498ebc" (UID: "5e1c2b14-07f4-4150-8bbf-6114e3498ebc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:25:14.811711 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:14.811639 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1c2b14-07f4-4150-8bbf-6114e3498ebc-maas-api-service-ca\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:25:15.527460 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:15.527433 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9" Jul 16 13:25:15.527741 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:15.527425 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9" event={"ID":"5e1c2b14-07f4-4150-8bbf-6114e3498ebc","Type":"ContainerDied","Data":"a6e91c99fa213c9b48692d9581b67828385fbf9429bb95d9091cb3cc632ac5c5"} Jul 16 13:25:15.527741 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:15.527558 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6e91c99fa213c9b48692d9581b67828385fbf9429bb95d9091cb3cc632ac5c5" Jul 16 13:25:24.490819 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:24.490772 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" podUID="8f1f0152-be54-4b47-ae12-cbc80dafceb3" containerName="main" containerID="cri-o://3a2078873ccd51603c82e8f84738c18ec069a6475556a1c363cc5a982a89f36b" gracePeriod=45 Jul 16 13:25:24.755131 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:24.755070 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:25:24.910835 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:24.910793 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbkbl\" (UniqueName: \"kubernetes.io/projected/8f1f0152-be54-4b47-ae12-cbc80dafceb3-kube-api-access-xbkbl\") pod \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " Jul 16 13:25:24.910835 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:24.910844 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-tmp-dir\") pod \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " Jul 16 13:25:24.911103 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:24.910877 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f1f0152-be54-4b47-ae12-cbc80dafceb3-tls-certs\") pod \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " Jul 16 13:25:24.911103 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:24.910923 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-dshm\") pod \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " Jul 16 13:25:24.911103 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:24.910965 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-home\") pod \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " Jul 16 13:25:24.911103 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:24.911025 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-model-cache\") pod \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\" (UID: \"8f1f0152-be54-4b47-ae12-cbc80dafceb3\") " Jul 16 13:25:24.911346 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:24.911129 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f1f0152-be54-4b47-ae12-cbc80dafceb3" (UID: "8f1f0152-be54-4b47-ae12-cbc80dafceb3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:25:24.911346 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:24.911320 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-home" (OuterVolumeSpecName: "home") pod "8f1f0152-be54-4b47-ae12-cbc80dafceb3" (UID: "8f1f0152-be54-4b47-ae12-cbc80dafceb3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:25:24.911457 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:24.911390 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-tmp-dir\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:25:24.911457 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:24.911410 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-home\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:25:24.911561 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:24.911467 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f1f0152-be54-4b47-ae12-cbc80dafceb3" (UID: "8f1f0152-be54-4b47-ae12-cbc80dafceb3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:25:24.913384 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:24.913350 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-dshm" (OuterVolumeSpecName: "dshm") pod "8f1f0152-be54-4b47-ae12-cbc80dafceb3" (UID: "8f1f0152-be54-4b47-ae12-cbc80dafceb3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:25:24.913504 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:24.913435 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f1f0152-be54-4b47-ae12-cbc80dafceb3-kube-api-access-xbkbl" (OuterVolumeSpecName: "kube-api-access-xbkbl") pod "8f1f0152-be54-4b47-ae12-cbc80dafceb3" (UID: "8f1f0152-be54-4b47-ae12-cbc80dafceb3"). InnerVolumeSpecName "kube-api-access-xbkbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:25:24.913550 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:24.913518 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f1f0152-be54-4b47-ae12-cbc80dafceb3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f1f0152-be54-4b47-ae12-cbc80dafceb3" (UID: "8f1f0152-be54-4b47-ae12-cbc80dafceb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:25:25.012829 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:25.012748 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbkbl\" (UniqueName: \"kubernetes.io/projected/8f1f0152-be54-4b47-ae12-cbc80dafceb3-kube-api-access-xbkbl\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:25:25.012829 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:25.012775 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f1f0152-be54-4b47-ae12-cbc80dafceb3-tls-certs\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:25:25.012829 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:25.012787 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-dshm\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:25:25.012829 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:25.012795 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f1f0152-be54-4b47-ae12-cbc80dafceb3-model-cache\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:25:25.583590 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:25.583558 2565 generic.go:356] "Generic (PLEG): container finished" podID="8f1f0152-be54-4b47-ae12-cbc80dafceb3" containerID="3a2078873ccd51603c82e8f84738c18ec069a6475556a1c363cc5a982a89f36b" exitCode=0 Jul 16 13:25:25.584013 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:25.583608 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" Jul 16 13:25:25.584013 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:25.583619 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" event={"ID":"8f1f0152-be54-4b47-ae12-cbc80dafceb3","Type":"ContainerDied","Data":"3a2078873ccd51603c82e8f84738c18ec069a6475556a1c363cc5a982a89f36b"} Jul 16 13:25:25.584013 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:25.583653 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4" event={"ID":"8f1f0152-be54-4b47-ae12-cbc80dafceb3","Type":"ContainerDied","Data":"0bc90b4b36c49e2aac9aba534693d337b348f0764fd624ee5e2878cf0c53482c"} Jul 16 13:25:25.584013 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:25.583668 2565 scope.go:117] "RemoveContainer" containerID="3a2078873ccd51603c82e8f84738c18ec069a6475556a1c363cc5a982a89f36b" Jul 16 13:25:25.593842 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:25.593825 2565 scope.go:117] "RemoveContainer" containerID="3a2078873ccd51603c82e8f84738c18ec069a6475556a1c363cc5a982a89f36b" Jul 16 13:25:25.594104 ip-10-0-138-198 kubenswrapper[2565]: E0716 13:25:25.594076 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a2078873ccd51603c82e8f84738c18ec069a6475556a1c363cc5a982a89f36b\": container with ID starting with 3a2078873ccd51603c82e8f84738c18ec069a6475556a1c363cc5a982a89f36b not found: ID does not exist" containerID="3a2078873ccd51603c82e8f84738c18ec069a6475556a1c363cc5a982a89f36b" Jul 16 13:25:25.594158 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:25.594104 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a2078873ccd51603c82e8f84738c18ec069a6475556a1c363cc5a982a89f36b"} err="failed to get container status \"3a2078873ccd51603c82e8f84738c18ec069a6475556a1c363cc5a982a89f36b\": rpc error: code = NotFound desc = could not find container \"3a2078873ccd51603c82e8f84738c18ec069a6475556a1c363cc5a982a89f36b\": container with ID starting with 3a2078873ccd51603c82e8f84738c18ec069a6475556a1c363cc5a982a89f36b not found: ID does not exist" Jul 16 13:25:25.609549 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:25.609523 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4"] Jul 16 13:25:25.614618 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:25.614596 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ab35b1/test-model-ab35b1-kserve-fb94dc7c-gkvq4"] Jul 16 13:25:26.545815 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:26.545779 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f1f0152-be54-4b47-ae12-cbc80dafceb3" path="/var/lib/kubelet/pods/8f1f0152-be54-4b47-ae12-cbc80dafceb3/volumes" Jul 16 13:25:35.747248 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:35.747209 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq"] Jul 16 13:25:35.747631 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:35.747584 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f1f0152-be54-4b47-ae12-cbc80dafceb3" containerName="main" Jul 16 13:25:35.747631 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:35.747596 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f1f0152-be54-4b47-ae12-cbc80dafceb3" containerName="main" Jul 16 13:25:35.747631 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:35.747606 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e1c2b14-07f4-4150-8bbf-6114e3498ebc" containerName="revoke-keys" Jul 16 13:25:35.747631 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:35.747611 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e1c2b14-07f4-4150-8bbf-6114e3498ebc" containerName="revoke-keys" Jul 16 13:25:35.747764 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:35.747678 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f1f0152-be54-4b47-ae12-cbc80dafceb3" containerName="main" Jul 16 13:25:35.747764 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:35.747686 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e1c2b14-07f4-4150-8bbf-6114e3498ebc" containerName="revoke-keys" Jul 16 13:25:35.750683 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:35.750667 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq" Jul 16 13:25:35.753095 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:35.753063 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9kff4\"" Jul 16 13:25:35.753221 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:35.753078 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 13:25:35.759373 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:35.759349 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq"] Jul 16 13:25:35.917350 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:35.917311 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83dcf67-ae55-40a6-8883-d624b7050ed5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq\" (UID: \"f83dcf67-ae55-40a6-8883-d624b7050ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq" Jul 16 13:25:36.018389 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:36.018364 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83dcf67-ae55-40a6-8883-d624b7050ed5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq\" (UID: \"f83dcf67-ae55-40a6-8883-d624b7050ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq" Jul 16 13:25:36.019054 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:36.019028 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83dcf67-ae55-40a6-8883-d624b7050ed5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq\" (UID: \"f83dcf67-ae55-40a6-8883-d624b7050ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq" Jul 16 13:25:36.062212 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:36.062178 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq" Jul 16 13:25:36.202413 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:36.202382 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq"] Jul 16 13:25:36.203631 ip-10-0-138-198 kubenswrapper[2565]: W0716 13:25:36.203601 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83dcf67_ae55_40a6_8883_d624b7050ed5.slice/crio-4ebc37bc5dd1de777422eef94ca1afce5afd295c2f7a5083c1cff1c77ca83667 WatchSource:0}: Error finding container 4ebc37bc5dd1de777422eef94ca1afce5afd295c2f7a5083c1cff1c77ca83667: Status 404 returned error can't find the container with id 4ebc37bc5dd1de777422eef94ca1afce5afd295c2f7a5083c1cff1c77ca83667 Jul 16 13:25:36.647377 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:36.647334 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq" event={"ID":"f83dcf67-ae55-40a6-8883-d624b7050ed5","Type":"ContainerStarted","Data":"1fdf75cd428fce36d0e5a50842ae3fd7ffed9dd8683b5586906a521905e44109"} Jul 16 13:25:36.647542 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:36.647395 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq" event={"ID":"f83dcf67-ae55-40a6-8883-d624b7050ed5","Type":"ContainerStarted","Data":"4ebc37bc5dd1de777422eef94ca1afce5afd295c2f7a5083c1cff1c77ca83667"} Jul 16 13:25:36.667648 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:36.667602 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq" podStartSLOduration=1.6675867709999999 podStartE2EDuration="1.667586771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:25:36.665151248 +0000 UTC m=+2444.660292013" watchObservedRunningTime="2026-07-16 13:25:36.667586771 +0000 UTC m=+2444.662727531" Jul 16 13:25:37.655479 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:37.655448 2565 generic.go:356] "Generic (PLEG): container finished" podID="f83dcf67-ae55-40a6-8883-d624b7050ed5" containerID="1fdf75cd428fce36d0e5a50842ae3fd7ffed9dd8683b5586906a521905e44109" exitCode=0 Jul 16 13:25:37.655868 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:37.655495 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq" event={"ID":"f83dcf67-ae55-40a6-8883-d624b7050ed5","Type":"ContainerDied","Data":"1fdf75cd428fce36d0e5a50842ae3fd7ffed9dd8683b5586906a521905e44109"} Jul 16 13:25:38.793864 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:38.793843 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq" Jul 16 13:25:38.945437 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:38.945340 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83dcf67-ae55-40a6-8883-d624b7050ed5-maas-api-service-ca\") pod \"f83dcf67-ae55-40a6-8883-d624b7050ed5\" (UID: \"f83dcf67-ae55-40a6-8883-d624b7050ed5\") " Jul 16 13:25:38.945705 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:38.945682 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f83dcf67-ae55-40a6-8883-d624b7050ed5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f83dcf67-ae55-40a6-8883-d624b7050ed5" (UID: "f83dcf67-ae55-40a6-8883-d624b7050ed5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:25:39.046987 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:39.046952 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83dcf67-ae55-40a6-8883-d624b7050ed5-maas-api-service-ca\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:25:39.668035 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:39.668001 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq" event={"ID":"f83dcf67-ae55-40a6-8883-d624b7050ed5","Type":"ContainerDied","Data":"4ebc37bc5dd1de777422eef94ca1afce5afd295c2f7a5083c1cff1c77ca83667"} Jul 16 13:25:39.668204 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:39.668049 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ebc37bc5dd1de777422eef94ca1afce5afd295c2f7a5083c1cff1c77ca83667" Jul 16 13:25:39.668204 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:25:39.668014 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq" Jul 16 13:26:26.441174 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:26.441138 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-62a003-v2fgs"] Jul 16 13:26:26.441713 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:26.441549 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f83dcf67-ae55-40a6-8883-d624b7050ed5" containerName="revoke-keys" Jul 16 13:26:26.441713 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:26.441561 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f83dcf67-ae55-40a6-8883-d624b7050ed5" containerName="revoke-keys" Jul 16 13:26:26.441713 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:26.441636 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f83dcf67-ae55-40a6-8883-d624b7050ed5" containerName="revoke-keys" Jul 16 13:26:26.444801 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:26.444785 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-62a003-v2fgs" Jul 16 13:26:26.447111 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:26.447086 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 13:26:26.447252 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:26.447096 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9kff4\"" Jul 16 13:26:26.453162 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:26.452918 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-62a003-v2fgs"] Jul 16 13:26:26.479562 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:26.479528 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180f2e64-dad0-430a-948f-21fa3e82cca0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-62a003-v2fgs\" (UID: \"180f2e64-dad0-430a-948f-21fa3e82cca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-62a003-v2fgs" Jul 16 13:26:26.580923 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:26.580893 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180f2e64-dad0-430a-948f-21fa3e82cca0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-62a003-v2fgs\" (UID: \"180f2e64-dad0-430a-948f-21fa3e82cca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-62a003-v2fgs" Jul 16 13:26:26.581560 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:26.581540 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180f2e64-dad0-430a-948f-21fa3e82cca0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-62a003-v2fgs\" (UID: \"180f2e64-dad0-430a-948f-21fa3e82cca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-62a003-v2fgs" Jul 16 13:26:26.757796 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:26.757707 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-62a003-v2fgs" Jul 16 13:26:26.898888 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:26.898859 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-62a003-v2fgs"] Jul 16 13:26:26.900333 ip-10-0-138-198 kubenswrapper[2565]: W0716 13:26:26.900304 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180f2e64_dad0_430a_948f_21fa3e82cca0.slice/crio-3347fe00d44a6c994ae7cd20295191058afaa794b4ebe89d3040395cea5ccc75 WatchSource:0}: Error finding container 3347fe00d44a6c994ae7cd20295191058afaa794b4ebe89d3040395cea5ccc75: Status 404 returned error can't find the container with id 3347fe00d44a6c994ae7cd20295191058afaa794b4ebe89d3040395cea5ccc75 Jul 16 13:26:26.940027 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:26.939997 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-62a003-v2fgs" event={"ID":"180f2e64-dad0-430a-948f-21fa3e82cca0","Type":"ContainerStarted","Data":"3347fe00d44a6c994ae7cd20295191058afaa794b4ebe89d3040395cea5ccc75"} Jul 16 13:26:27.947585 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:27.947550 2565 generic.go:356] "Generic (PLEG): container finished" podID="180f2e64-dad0-430a-948f-21fa3e82cca0" containerID="01199e1cd58b4e096387ebe2ddd36dc8f0672f1bbe484e180dbf5daf83b9fb9a" exitCode=0 Jul 16 13:26:27.948018 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:27.947638 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-62a003-v2fgs" event={"ID":"180f2e64-dad0-430a-948f-21fa3e82cca0","Type":"ContainerDied","Data":"01199e1cd58b4e096387ebe2ddd36dc8f0672f1bbe484e180dbf5daf83b9fb9a"} Jul 16 13:26:29.089520 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:29.089496 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-62a003-v2fgs" Jul 16 13:26:29.099500 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:29.099479 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180f2e64-dad0-430a-948f-21fa3e82cca0-maas-api-service-ca\") pod \"180f2e64-dad0-430a-948f-21fa3e82cca0\" (UID: \"180f2e64-dad0-430a-948f-21fa3e82cca0\") " Jul 16 13:26:29.099779 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:29.099761 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180f2e64-dad0-430a-948f-21fa3e82cca0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "180f2e64-dad0-430a-948f-21fa3e82cca0" (UID: "180f2e64-dad0-430a-948f-21fa3e82cca0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:26:29.200668 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:29.200632 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180f2e64-dad0-430a-948f-21fa3e82cca0-maas-api-service-ca\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:26:29.959657 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:29.959627 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-62a003-v2fgs" Jul 16 13:26:29.959805 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:29.959669 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-62a003-v2fgs" event={"ID":"180f2e64-dad0-430a-948f-21fa3e82cca0","Type":"ContainerDied","Data":"3347fe00d44a6c994ae7cd20295191058afaa794b4ebe89d3040395cea5ccc75"} Jul 16 13:26:29.959805 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:29.959715 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3347fe00d44a6c994ae7cd20295191058afaa794b4ebe89d3040395cea5ccc75" Jul 16 13:26:52.603073 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:52.602991 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv"] Jul 16 13:26:52.603445 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:52.603394 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="180f2e64-dad0-430a-948f-21fa3e82cca0" containerName="revoke-keys" Jul 16 13:26:52.603445 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:52.603406 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="180f2e64-dad0-430a-948f-21fa3e82cca0" containerName="revoke-keys" Jul 16 13:26:52.603518 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:52.603462 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="180f2e64-dad0-430a-948f-21fa3e82cca0" containerName="revoke-keys" Jul 16 13:26:52.606689 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:52.606668 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv" Jul 16 13:26:52.609113 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:52.609098 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 13:26:52.609183 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:52.609117 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9kff4\"" Jul 16 13:26:52.615250 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:52.615207 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv"] Jul 16 13:26:52.700478 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:52.700442 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39e2b07-cea0-46f6-b398-a39e3a901445-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv\" (UID: \"f39e2b07-cea0-46f6-b398-a39e3a901445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv" Jul 16 13:26:52.801395 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:52.801364 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39e2b07-cea0-46f6-b398-a39e3a901445-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv\" (UID: \"f39e2b07-cea0-46f6-b398-a39e3a901445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv" Jul 16 13:26:52.802012 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:52.801994 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39e2b07-cea0-46f6-b398-a39e3a901445-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv\" (UID: \"f39e2b07-cea0-46f6-b398-a39e3a901445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv" Jul 16 13:26:52.919386 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:52.919315 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv" Jul 16 13:26:53.059254 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:53.059211 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv"] Jul 16 13:26:53.060813 ip-10-0-138-198 kubenswrapper[2565]: W0716 13:26:53.060786 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39e2b07_cea0_46f6_b398_a39e3a901445.slice/crio-100cbd6a3bdc48c00f3212b33f46e744dbde37fb9875178c1446ca7ccffaee48 WatchSource:0}: Error finding container 100cbd6a3bdc48c00f3212b33f46e744dbde37fb9875178c1446ca7ccffaee48: Status 404 returned error can't find the container with id 100cbd6a3bdc48c00f3212b33f46e744dbde37fb9875178c1446ca7ccffaee48 Jul 16 13:26:53.091497 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:53.091470 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv" event={"ID":"f39e2b07-cea0-46f6-b398-a39e3a901445","Type":"ContainerStarted","Data":"100cbd6a3bdc48c00f3212b33f46e744dbde37fb9875178c1446ca7ccffaee48"} Jul 16 13:26:54.098311 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:54.098272 2565 generic.go:356] "Generic (PLEG): container finished" podID="f39e2b07-cea0-46f6-b398-a39e3a901445" containerID="c3c961b58471b78fa1c38a4612d247d538221aed2ad58ffe8f5e420fb435a040" exitCode=0 Jul 16 13:26:54.098717 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:54.098358 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv" event={"ID":"f39e2b07-cea0-46f6-b398-a39e3a901445","Type":"ContainerDied","Data":"c3c961b58471b78fa1c38a4612d247d538221aed2ad58ffe8f5e420fb435a040"} Jul 16 13:26:55.247893 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:55.247870 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv" Jul 16 13:26:55.324509 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:55.324477 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39e2b07-cea0-46f6-b398-a39e3a901445-maas-api-service-ca\") pod \"f39e2b07-cea0-46f6-b398-a39e3a901445\" (UID: \"f39e2b07-cea0-46f6-b398-a39e3a901445\") " Jul 16 13:26:55.324914 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:55.324893 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f39e2b07-cea0-46f6-b398-a39e3a901445-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f39e2b07-cea0-46f6-b398-a39e3a901445" (UID: "f39e2b07-cea0-46f6-b398-a39e3a901445"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:26:55.425534 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:55.425449 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39e2b07-cea0-46f6-b398-a39e3a901445-maas-api-service-ca\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:26:56.113005 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:56.112971 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv" event={"ID":"f39e2b07-cea0-46f6-b398-a39e3a901445","Type":"ContainerDied","Data":"100cbd6a3bdc48c00f3212b33f46e744dbde37fb9875178c1446ca7ccffaee48"} Jul 16 13:26:56.113005 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:56.112999 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv" Jul 16 13:26:56.113284 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:26:56.113017 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="100cbd6a3bdc48c00f3212b33f46e744dbde37fb9875178c1446ca7ccffaee48" Jul 16 13:27:27.384866 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.384829 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9"] Jul 16 13:27:27.385332 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.385223 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f39e2b07-cea0-46f6-b398-a39e3a901445" containerName="revoke-keys" Jul 16 13:27:27.385332 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.385254 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f39e2b07-cea0-46f6-b398-a39e3a901445" containerName="revoke-keys" Jul 16 13:27:27.385332 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.385332 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f39e2b07-cea0-46f6-b398-a39e3a901445" containerName="revoke-keys" Jul 16 13:27:27.388552 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.388536 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.390841 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.390823 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-c7dd05-openshift-default-dockercfg-chxkn\"" Jul 16 13:27:27.404353 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.404329 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9"] Jul 16 13:27:27.410788 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.410757 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-workload-socket\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.410938 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.410809 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-credential-socket\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.410938 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.410870 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bb0969-5576-481a-b2d5-60a08a9d39de-istiod-ca-cert\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.410938 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.410904 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-envoy\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.410938 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.410922 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-data\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.411088 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.410942 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-podinfo\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.411088 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.411027 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq5hs\" (UniqueName: \"kubernetes.io/projected/e3bb0969-5576-481a-b2d5-60a08a9d39de-kube-api-access-vq5hs\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.411088 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.411074 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-token\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.411187 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.411105 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-workload-certs\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.512290 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.512256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bb0969-5576-481a-b2d5-60a08a9d39de-istiod-ca-cert\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.512507 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.512304 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-envoy\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.512507 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.512332 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-data\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.512507 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.512362 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-podinfo\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.512507 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.512407 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq5hs\" (UniqueName: \"kubernetes.io/projected/e3bb0969-5576-481a-b2d5-60a08a9d39de-kube-api-access-vq5hs\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.512507 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.512459 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-token\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.512507 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.512495 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-workload-certs\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.512840 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.512527 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-workload-socket\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.512840 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.512571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-credential-socket\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.512940 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.512853 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-data\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.512940 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.512905 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-credential-socket\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.513216 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.513023 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-workload-certs\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.513216 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.513072 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-workload-socket\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.513216 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.513137 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bb0969-5576-481a-b2d5-60a08a9d39de-istiod-ca-cert\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.514875 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.514850 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-envoy\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.515047 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.515029 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-podinfo\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.521703 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.521681 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-token\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.521864 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.521846 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq5hs\" (UniqueName: \"kubernetes.io/projected/e3bb0969-5576-481a-b2d5-60a08a9d39de-kube-api-access-vq5hs\") pod \"e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.699488 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.699389 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:27.856403 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.856374 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9"] Jul 16 13:27:27.857442 ip-10-0-138-198 kubenswrapper[2565]: W0716 13:27:27.857415 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3bb0969_5576_481a_b2d5_60a08a9d39de.slice/crio-e72e380de8708eec8b1b7f9b0ded7659dcf5aa0c77076a1969764d7a596e20d9 WatchSource:0}: Error finding container e72e380de8708eec8b1b7f9b0ded7659dcf5aa0c77076a1969764d7a596e20d9: Status 404 returned error can't find the container with id e72e380de8708eec8b1b7f9b0ded7659dcf5aa0c77076a1969764d7a596e20d9 Jul 16 13:27:27.859685 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.859644 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 16 13:27:27.859792 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.859734 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 16 13:27:27.859792 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:27.859784 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 16 13:27:28.307055 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:28.307019 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" event={"ID":"e3bb0969-5576-481a-b2d5-60a08a9d39de","Type":"ContainerStarted","Data":"e8599fd23d7a0f21cb9cdcb7074874b98048858adf998f6cad30ba4e3bd4d8fe"} Jul 16 13:27:28.307252 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:28.307071 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" event={"ID":"e3bb0969-5576-481a-b2d5-60a08a9d39de","Type":"ContainerStarted","Data":"e72e380de8708eec8b1b7f9b0ded7659dcf5aa0c77076a1969764d7a596e20d9"} Jul 16 13:27:28.327955 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:28.327911 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" podStartSLOduration=1.327894668 podStartE2EDuration="1.327894668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:27:28.326842916 +0000 UTC m=+2556.321983703" watchObservedRunningTime="2026-07-16 13:27:28.327894668 +0000 UTC m=+2556.323035428" Jul 16 13:27:28.699716 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:28.699637 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:28.705089 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:28.705065 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:29.312608 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:29.312567 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:29.313586 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:29.313564 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:43.400581 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:43.400549 2565 generic.go:356] "Generic (PLEG): container finished" podID="e3bb0969-5576-481a-b2d5-60a08a9d39de" containerID="e8599fd23d7a0f21cb9cdcb7074874b98048858adf998f6cad30ba4e3bd4d8fe" exitCode=0 Jul 16 13:27:43.401125 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:43.400631 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" event={"ID":"e3bb0969-5576-481a-b2d5-60a08a9d39de","Type":"ContainerDied","Data":"e8599fd23d7a0f21cb9cdcb7074874b98048858adf998f6cad30ba4e3bd4d8fe"} Jul 16 13:27:43.401203 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:43.401130 2565 scope.go:117] "RemoveContainer" containerID="e8599fd23d7a0f21cb9cdcb7074874b98048858adf998f6cad30ba4e3bd4d8fe" Jul 16 13:27:43.402164 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:43.402136 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 16 13:27:43.402287 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:43.402202 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 16 13:27:43.402287 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:43.402253 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 16 13:27:44.313469 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:44.313437 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:44.408849 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:44.408810 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" event={"ID":"e3bb0969-5576-481a-b2d5-60a08a9d39de","Type":"ContainerStarted","Data":"d21888187d1675d39a9d7018415d6b63e11c52afc08ce03c9b0c235105ff70a4"} Jul 16 13:27:44.700076 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:44.699981 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:44.701258 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:44.701208 2565 patch_prober.go:28] interesting pod/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 16 13:27:44.701400 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:44.701281 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" podUID="e3bb0969-5576-481a-b2d5-60a08a9d39de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 16 13:27:45.700457 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:45.700425 2565 patch_prober.go:28] interesting pod/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 16 13:27:45.700845 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:45.700494 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" podUID="e3bb0969-5576-481a-b2d5-60a08a9d39de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 16 13:27:46.700077 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:46.700042 2565 patch_prober.go:28] interesting pod/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 16 13:27:46.700297 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:46.700093 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" podUID="e3bb0969-5576-481a-b2d5-60a08a9d39de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 16 13:27:47.699893 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:47.699853 2565 patch_prober.go:28] interesting pod/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 16 13:27:47.700317 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:47.699909 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" podUID="e3bb0969-5576-481a-b2d5-60a08a9d39de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 16 13:27:48.703877 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:48.703837 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:48.704281 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:48.704103 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:27:48.704899 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:27:48.704881 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:28:11.680888 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.680856 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw"] Jul 16 13:28:11.684428 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.684413 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.687166 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.687137 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c7dd05\"/\"openshift-service-ca.crt\"" Jul 16 13:28:11.687435 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.687417 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c7dd05\"/\"kube-root-ca.crt\"" Jul 16 13:28:11.688313 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.688297 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c7dd05\"/\"default-dockercfg-hpcfq\"" Jul 16 13:28:11.688409 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.688364 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c7dd05\"/\"auth-test-model-c7dd05-kserve-self-signed-certs\"" Jul 16 13:28:11.704841 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.704817 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw"] Jul 16 13:28:11.828153 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.828121 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-home\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.828357 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.828163 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-tmp-dir\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.828357 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.828189 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4987643-36b1-4e9d-8aaf-4d10b4500da0-tls-certs\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.828357 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.828215 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-dshm\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.828357 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.828288 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-model-cache\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.828357 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.828323 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4mmr\" (UniqueName: \"kubernetes.io/projected/f4987643-36b1-4e9d-8aaf-4d10b4500da0-kube-api-access-b4mmr\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.929433 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.929400 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-home\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.929573 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.929442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-tmp-dir\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.929573 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.929471 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4987643-36b1-4e9d-8aaf-4d10b4500da0-tls-certs\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.929573 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.929493 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-dshm\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.929573 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.929531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-model-cache\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.929573 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.929570 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4mmr\" (UniqueName: \"kubernetes.io/projected/f4987643-36b1-4e9d-8aaf-4d10b4500da0-kube-api-access-b4mmr\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.929820 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.929788 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-home\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.929882 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.929859 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-tmp-dir\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.929993 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.929969 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-model-cache\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.931981 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.931918 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-dshm\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.932145 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.932128 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4987643-36b1-4e9d-8aaf-4d10b4500da0-tls-certs\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.941048 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.941019 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4mmr\" (UniqueName: \"kubernetes.io/projected/f4987643-36b1-4e9d-8aaf-4d10b4500da0-kube-api-access-b4mmr\") pod \"auth-test-model-c7dd05-kserve-658dd545f5-2kvnw\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:11.993779 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:11.993733 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:12.133633 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:12.133611 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw"] Jul 16 13:28:12.135681 ip-10-0-138-198 kubenswrapper[2565]: W0716 13:28:12.135654 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4987643_36b1_4e9d_8aaf_4d10b4500da0.slice/crio-b4e64621e166d61aa68860e40231bd4f8357352b620c58e8226da26c13592be6 WatchSource:0}: Error finding container b4e64621e166d61aa68860e40231bd4f8357352b620c58e8226da26c13592be6: Status 404 returned error can't find the container with id b4e64621e166d61aa68860e40231bd4f8357352b620c58e8226da26c13592be6 Jul 16 13:28:12.573507 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:12.573472 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" event={"ID":"f4987643-36b1-4e9d-8aaf-4d10b4500da0","Type":"ContainerStarted","Data":"8a662458270a519970d417348dc230a376712d46c5afb39a75935c3bfb8bef9f"} Jul 16 13:28:12.573691 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:12.573513 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" event={"ID":"f4987643-36b1-4e9d-8aaf-4d10b4500da0","Type":"ContainerStarted","Data":"b4e64621e166d61aa68860e40231bd4f8357352b620c58e8226da26c13592be6"} Jul 16 13:28:12.602284 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:12.602219 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" podStartSLOduration=1.602203445 podStartE2EDuration="1.602203445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:28:12.59947909 +0000 UTC m=+2600.594619843" watchObservedRunningTime="2026-07-16 13:28:12.602203445 +0000 UTC m=+2600.597344205" Jul 16 13:28:21.994133 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:21.994049 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:21.994587 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:21.994397 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:22.006614 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:22.006591 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:28:22.642288 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:28:22.642256 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:29:52.739255 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:29:52.739134 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 13:29:52.747665 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:29:52.747645 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 13:30:22.377426 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:22.377390 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw"] Jul 16 13:30:30.070897 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.070863 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6"] Jul 16 13:30:30.080297 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.080275 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.083463 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.083439 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-978d92\"/\"kube-root-ca.crt\"" Jul 16 13:30:30.083594 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.083460 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-978d92\"/\"openshift-service-ca.crt\"" Jul 16 13:30:30.083594 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.083460 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-978d92\"/\"default-dockercfg-zlwnw\"" Jul 16 13:30:30.083724 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.083524 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-978d92\"/\"sub-test-model-978d92-kserve-self-signed-certs\"" Jul 16 13:30:30.085567 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.085541 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6"] Jul 16 13:30:30.143754 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.143731 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-model-cache\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.143894 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.143770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-tmp-dir\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.143894 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.143789 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-479x7\" (UniqueName: \"kubernetes.io/projected/33b35b53-6779-433e-a68a-6af8be6ed600-kube-api-access-479x7\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.143894 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.143879 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-home\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.143999 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.143955 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b35b53-6779-433e-a68a-6af8be6ed600-tls-certs\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.143999 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.143983 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-dshm\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.244421 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.244389 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b35b53-6779-433e-a68a-6af8be6ed600-tls-certs\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.244582 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.244433 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-dshm\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.244582 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.244468 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-model-cache\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.244582 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.244501 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-tmp-dir\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.244582 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.244525 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479x7\" (UniqueName: \"kubernetes.io/projected/33b35b53-6779-433e-a68a-6af8be6ed600-kube-api-access-479x7\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.244773 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.244582 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-home\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.244906 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.244877 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-model-cache\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.245013 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.244969 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-tmp-dir\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.245108 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.245086 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-home\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.247083 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.247061 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b35b53-6779-433e-a68a-6af8be6ed600-tls-certs\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.247163 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.247117 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-dshm\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.254183 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.254157 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-479x7\" (UniqueName: \"kubernetes.io/projected/33b35b53-6779-433e-a68a-6af8be6ed600-kube-api-access-479x7\") pod \"sub-test-model-978d92-kserve-6cb997c789-55gb6\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.391712 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.391616 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:30.538601 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:30.538561 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6"] Jul 16 13:30:30.539883 ip-10-0-138-198 kubenswrapper[2565]: W0716 13:30:30.539852 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33b35b53_6779_433e_a68a_6af8be6ed600.slice/crio-572afe30bf2c9ab6e09b5174a87cd8fab4d3fc5d561865795e884a205e4ef80e WatchSource:0}: Error finding container 572afe30bf2c9ab6e09b5174a87cd8fab4d3fc5d561865795e884a205e4ef80e: Status 404 returned error can't find the container with id 572afe30bf2c9ab6e09b5174a87cd8fab4d3fc5d561865795e884a205e4ef80e Jul 16 13:30:31.377113 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:31.377081 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" event={"ID":"33b35b53-6779-433e-a68a-6af8be6ed600","Type":"ContainerStarted","Data":"ab7c23dfd10a6dbf9e81d28bf7ee27355f54868936e88d1dc12d0e32c102c489"} Jul 16 13:30:31.377470 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:31.377122 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" event={"ID":"33b35b53-6779-433e-a68a-6af8be6ed600","Type":"ContainerStarted","Data":"572afe30bf2c9ab6e09b5174a87cd8fab4d3fc5d561865795e884a205e4ef80e"} Jul 16 13:30:31.398628 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:31.398578 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" podStartSLOduration=1.398563195 podStartE2EDuration="1.398563195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:30:31.397967747 +0000 UTC m=+2739.393108507" watchObservedRunningTime="2026-07-16 13:30:31.398563195 +0000 UTC m=+2739.393703985" Jul 16 13:30:37.397837 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.397788 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" podUID="f4987643-36b1-4e9d-8aaf-4d10b4500da0" containerName="main" containerID="cri-o://8a662458270a519970d417348dc230a376712d46c5afb39a75935c3bfb8bef9f" gracePeriod=45 Jul 16 13:30:37.656180 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.656118 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:30:37.715697 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.715669 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-model-cache\") pod \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " Jul 16 13:30:37.715882 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.715720 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4mmr\" (UniqueName: \"kubernetes.io/projected/f4987643-36b1-4e9d-8aaf-4d10b4500da0-kube-api-access-b4mmr\") pod \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " Jul 16 13:30:37.715882 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.715747 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4987643-36b1-4e9d-8aaf-4d10b4500da0-tls-certs\") pod \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " Jul 16 13:30:37.715882 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.715771 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-tmp-dir\") pod \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " Jul 16 13:30:37.715882 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.715787 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-dshm\") pod \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " Jul 16 13:30:37.715882 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.715810 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-home\") pod \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\" (UID: \"f4987643-36b1-4e9d-8aaf-4d10b4500da0\") " Jul 16 13:30:37.716153 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.715983 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4987643-36b1-4e9d-8aaf-4d10b4500da0" (UID: "f4987643-36b1-4e9d-8aaf-4d10b4500da0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:30:37.716153 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.716011 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4987643-36b1-4e9d-8aaf-4d10b4500da0" (UID: "f4987643-36b1-4e9d-8aaf-4d10b4500da0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:30:37.716153 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.716134 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-tmp-dir\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:30:37.716315 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.716153 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-model-cache\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:30:37.716315 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.716254 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-home" (OuterVolumeSpecName: "home") pod "f4987643-36b1-4e9d-8aaf-4d10b4500da0" (UID: "f4987643-36b1-4e9d-8aaf-4d10b4500da0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:30:37.718283 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.718252 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4987643-36b1-4e9d-8aaf-4d10b4500da0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4987643-36b1-4e9d-8aaf-4d10b4500da0" (UID: "f4987643-36b1-4e9d-8aaf-4d10b4500da0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:30:37.718283 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.718223 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-dshm" (OuterVolumeSpecName: "dshm") pod "f4987643-36b1-4e9d-8aaf-4d10b4500da0" (UID: "f4987643-36b1-4e9d-8aaf-4d10b4500da0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:30:37.718419 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.718316 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4987643-36b1-4e9d-8aaf-4d10b4500da0-kube-api-access-b4mmr" (OuterVolumeSpecName: "kube-api-access-b4mmr") pod "f4987643-36b1-4e9d-8aaf-4d10b4500da0" (UID: "f4987643-36b1-4e9d-8aaf-4d10b4500da0"). InnerVolumeSpecName "kube-api-access-b4mmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:30:37.817353 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.817321 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-dshm\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:30:37.817353 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.817348 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4987643-36b1-4e9d-8aaf-4d10b4500da0-home\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:30:37.817353 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.817356 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4mmr\" (UniqueName: \"kubernetes.io/projected/f4987643-36b1-4e9d-8aaf-4d10b4500da0-kube-api-access-b4mmr\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:30:37.817582 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:37.817366 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4987643-36b1-4e9d-8aaf-4d10b4500da0-tls-certs\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:30:38.419899 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:38.419858 2565 generic.go:356] "Generic (PLEG): container finished" podID="f4987643-36b1-4e9d-8aaf-4d10b4500da0" containerID="8a662458270a519970d417348dc230a376712d46c5afb39a75935c3bfb8bef9f" exitCode=0 Jul 16 13:30:38.419899 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:38.419898 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" Jul 16 13:30:38.420417 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:38.419949 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" event={"ID":"f4987643-36b1-4e9d-8aaf-4d10b4500da0","Type":"ContainerDied","Data":"8a662458270a519970d417348dc230a376712d46c5afb39a75935c3bfb8bef9f"} Jul 16 13:30:38.420417 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:38.419991 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw" event={"ID":"f4987643-36b1-4e9d-8aaf-4d10b4500da0","Type":"ContainerDied","Data":"b4e64621e166d61aa68860e40231bd4f8357352b620c58e8226da26c13592be6"} Jul 16 13:30:38.420417 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:38.420008 2565 scope.go:117] "RemoveContainer" containerID="8a662458270a519970d417348dc230a376712d46c5afb39a75935c3bfb8bef9f" Jul 16 13:30:38.433508 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:38.433486 2565 scope.go:117] "RemoveContainer" containerID="8a662458270a519970d417348dc230a376712d46c5afb39a75935c3bfb8bef9f" Jul 16 13:30:38.433822 ip-10-0-138-198 kubenswrapper[2565]: E0716 13:30:38.433801 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a662458270a519970d417348dc230a376712d46c5afb39a75935c3bfb8bef9f\": container with ID starting with 8a662458270a519970d417348dc230a376712d46c5afb39a75935c3bfb8bef9f not found: ID does not exist" containerID="8a662458270a519970d417348dc230a376712d46c5afb39a75935c3bfb8bef9f" Jul 16 13:30:38.433874 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:38.433833 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a662458270a519970d417348dc230a376712d46c5afb39a75935c3bfb8bef9f"} err="failed to get container status \"8a662458270a519970d417348dc230a376712d46c5afb39a75935c3bfb8bef9f\": rpc error: code = NotFound desc = could not find container \"8a662458270a519970d417348dc230a376712d46c5afb39a75935c3bfb8bef9f\": container with ID starting with 8a662458270a519970d417348dc230a376712d46c5afb39a75935c3bfb8bef9f not found: ID does not exist" Jul 16 13:30:38.448566 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:38.448529 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw"] Jul 16 13:30:38.454499 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:38.454476 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/auth-test-model-c7dd05-kserve-658dd545f5-2kvnw"] Jul 16 13:30:38.546157 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:38.546127 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4987643-36b1-4e9d-8aaf-4d10b4500da0" path="/var/lib/kubelet/pods/f4987643-36b1-4e9d-8aaf-4d10b4500da0/volumes" Jul 16 13:30:40.391761 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:40.391719 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:40.391761 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:40.391763 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:40.404127 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:40.404102 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:30:40.443369 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:30:40.443343 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:31:34.975392 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:34.975357 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6"] Jul 16 13:31:49.995970 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:49.995920 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" podUID="33b35b53-6779-433e-a68a-6af8be6ed600" containerName="main" containerID="cri-o://ab7c23dfd10a6dbf9e81d28bf7ee27355f54868936e88d1dc12d0e32c102c489" gracePeriod=45 Jul 16 13:31:50.249627 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.249569 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:31:50.291169 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.291142 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-model-cache\") pod \"33b35b53-6779-433e-a68a-6af8be6ed600\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " Jul 16 13:31:50.291346 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.291199 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-tmp-dir\") pod \"33b35b53-6779-433e-a68a-6af8be6ed600\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " Jul 16 13:31:50.291346 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.291283 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-dshm\") pod \"33b35b53-6779-433e-a68a-6af8be6ed600\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " Jul 16 13:31:50.291346 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.291314 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b35b53-6779-433e-a68a-6af8be6ed600-tls-certs\") pod \"33b35b53-6779-433e-a68a-6af8be6ed600\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " Jul 16 13:31:50.291346 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.291349 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-479x7\" (UniqueName: \"kubernetes.io/projected/33b35b53-6779-433e-a68a-6af8be6ed600-kube-api-access-479x7\") pod \"33b35b53-6779-433e-a68a-6af8be6ed600\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " Jul 16 13:31:50.291571 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.291374 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-home\") pod \"33b35b53-6779-433e-a68a-6af8be6ed600\" (UID: \"33b35b53-6779-433e-a68a-6af8be6ed600\") " Jul 16 13:31:50.291571 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.291452 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-model-cache" (OuterVolumeSpecName: "model-cache") pod "33b35b53-6779-433e-a68a-6af8be6ed600" (UID: "33b35b53-6779-433e-a68a-6af8be6ed600"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:31:50.291571 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.291463 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33b35b53-6779-433e-a68a-6af8be6ed600" (UID: "33b35b53-6779-433e-a68a-6af8be6ed600"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:31:50.291714 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.291667 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-home" (OuterVolumeSpecName: "home") pod "33b35b53-6779-433e-a68a-6af8be6ed600" (UID: "33b35b53-6779-433e-a68a-6af8be6ed600"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:31:50.291769 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.291713 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-model-cache\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:31:50.291769 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.291731 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-tmp-dir\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:31:50.293879 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.293849 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b35b53-6779-433e-a68a-6af8be6ed600-kube-api-access-479x7" (OuterVolumeSpecName: "kube-api-access-479x7") pod "33b35b53-6779-433e-a68a-6af8be6ed600" (UID: "33b35b53-6779-433e-a68a-6af8be6ed600"). InnerVolumeSpecName "kube-api-access-479x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:31:50.293986 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.293885 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-dshm" (OuterVolumeSpecName: "dshm") pod "33b35b53-6779-433e-a68a-6af8be6ed600" (UID: "33b35b53-6779-433e-a68a-6af8be6ed600"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:31:50.294032 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.294000 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b35b53-6779-433e-a68a-6af8be6ed600-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33b35b53-6779-433e-a68a-6af8be6ed600" (UID: "33b35b53-6779-433e-a68a-6af8be6ed600"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:31:50.392130 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.392104 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-dshm\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:31:50.392130 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.392127 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b35b53-6779-433e-a68a-6af8be6ed600-tls-certs\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:31:50.392130 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.392137 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-479x7\" (UniqueName: \"kubernetes.io/projected/33b35b53-6779-433e-a68a-6af8be6ed600-kube-api-access-479x7\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:31:50.392363 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.392145 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b35b53-6779-433e-a68a-6af8be6ed600-home\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:31:50.833755 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.833655 2565 generic.go:356] "Generic (PLEG): container finished" podID="33b35b53-6779-433e-a68a-6af8be6ed600" containerID="ab7c23dfd10a6dbf9e81d28bf7ee27355f54868936e88d1dc12d0e32c102c489" exitCode=0 Jul 16 13:31:50.833755 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.833710 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" Jul 16 13:31:50.833982 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.833746 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" event={"ID":"33b35b53-6779-433e-a68a-6af8be6ed600","Type":"ContainerDied","Data":"ab7c23dfd10a6dbf9e81d28bf7ee27355f54868936e88d1dc12d0e32c102c489"} Jul 16 13:31:50.833982 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.833798 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6" event={"ID":"33b35b53-6779-433e-a68a-6af8be6ed600","Type":"ContainerDied","Data":"572afe30bf2c9ab6e09b5174a87cd8fab4d3fc5d561865795e884a205e4ef80e"} Jul 16 13:31:50.833982 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.833816 2565 scope.go:117] "RemoveContainer" containerID="ab7c23dfd10a6dbf9e81d28bf7ee27355f54868936e88d1dc12d0e32c102c489" Jul 16 13:31:50.842876 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.842861 2565 scope.go:117] "RemoveContainer" containerID="ab7c23dfd10a6dbf9e81d28bf7ee27355f54868936e88d1dc12d0e32c102c489" Jul 16 13:31:50.843142 ip-10-0-138-198 kubenswrapper[2565]: E0716 13:31:50.843120 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7c23dfd10a6dbf9e81d28bf7ee27355f54868936e88d1dc12d0e32c102c489\": container with ID starting with ab7c23dfd10a6dbf9e81d28bf7ee27355f54868936e88d1dc12d0e32c102c489 not found: ID does not exist" containerID="ab7c23dfd10a6dbf9e81d28bf7ee27355f54868936e88d1dc12d0e32c102c489" Jul 16 13:31:50.843264 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.843147 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7c23dfd10a6dbf9e81d28bf7ee27355f54868936e88d1dc12d0e32c102c489"} err="failed to get container status \"ab7c23dfd10a6dbf9e81d28bf7ee27355f54868936e88d1dc12d0e32c102c489\": rpc error: code = NotFound desc = could not find container \"ab7c23dfd10a6dbf9e81d28bf7ee27355f54868936e88d1dc12d0e32c102c489\": container with ID starting with ab7c23dfd10a6dbf9e81d28bf7ee27355f54868936e88d1dc12d0e32c102c489 not found: ID does not exist" Jul 16 13:31:50.866375 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.866354 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6"] Jul 16 13:31:50.872551 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:50.872531 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-978d92/sub-test-model-978d92-kserve-6cb997c789-55gb6"] Jul 16 13:31:52.551093 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:31:52.551052 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33b35b53-6779-433e-a68a-6af8be6ed600" path="/var/lib/kubelet/pods/33b35b53-6779-433e-a68a-6af8be6ed600/volumes" Jul 16 13:32:18.474834 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.474801 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d"] Jul 16 13:32:18.475298 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.475180 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b35b53-6779-433e-a68a-6af8be6ed600" containerName="main" Jul 16 13:32:18.475298 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.475189 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b35b53-6779-433e-a68a-6af8be6ed600" containerName="main" Jul 16 13:32:18.475298 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.475206 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4987643-36b1-4e9d-8aaf-4d10b4500da0" containerName="main" Jul 16 13:32:18.475298 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.475212 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4987643-36b1-4e9d-8aaf-4d10b4500da0" containerName="main" Jul 16 13:32:18.475298 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.475297 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4987643-36b1-4e9d-8aaf-4d10b4500da0" containerName="main" Jul 16 13:32:18.475487 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.475305 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b35b53-6779-433e-a68a-6af8be6ed600" containerName="main" Jul 16 13:32:18.478621 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.478600 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.481695 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.481672 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c7dd05\"/\"kube-root-ca.crt\"" Jul 16 13:32:18.482660 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.482637 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c7dd05\"/\"default-dockercfg-hpcfq\"" Jul 16 13:32:18.482757 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.482680 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c7dd05\"/\"openshift-service-ca.crt\"" Jul 16 13:32:18.482757 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.482680 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c7dd05\"/\"rate-test-model-c7dd05-kserve-self-signed-certs\"" Jul 16 13:32:18.496368 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.496347 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d"] Jul 16 13:32:18.544931 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.544889 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-dshm\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.545126 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.544956 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vhjb\" (UniqueName: \"kubernetes.io/projected/06e85349-db65-42c9-b5c6-ae14b9de175d-kube-api-access-2vhjb\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.545126 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.545000 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e85349-db65-42c9-b5c6-ae14b9de175d-tls-certs\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.545126 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.545031 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-home\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.545389 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.545134 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-model-cache\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.545389 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.545164 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-tmp-dir\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.646689 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.646650 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-model-cache\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.646689 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.646693 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-tmp-dir\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.646925 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.646728 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-dshm\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.646925 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.646748 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vhjb\" (UniqueName: \"kubernetes.io/projected/06e85349-db65-42c9-b5c6-ae14b9de175d-kube-api-access-2vhjb\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.646925 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.646771 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e85349-db65-42c9-b5c6-ae14b9de175d-tls-certs\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.646925 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.646786 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-home\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.647151 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.647127 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-model-cache\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.647219 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.647168 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-home\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.647299 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.647252 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-tmp-dir\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.649184 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.649162 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-dshm\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.649471 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.649450 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e85349-db65-42c9-b5c6-ae14b9de175d-tls-certs\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.658392 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.658370 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vhjb\" (UniqueName: \"kubernetes.io/projected/06e85349-db65-42c9-b5c6-ae14b9de175d-kube-api-access-2vhjb\") pod \"rate-test-model-c7dd05-kserve-5b46985547-n7r7d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.788299 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.788269 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:18.941951 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.941808 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d"] Jul 16 13:32:18.944319 ip-10-0-138-198 kubenswrapper[2565]: W0716 13:32:18.944295 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06e85349_db65_42c9_b5c6_ae14b9de175d.slice/crio-61d0c3b8f04f98e80718368264e97e47b63482552a3ddae2f6e5adb4dc3af1dc WatchSource:0}: Error finding container 61d0c3b8f04f98e80718368264e97e47b63482552a3ddae2f6e5adb4dc3af1dc: Status 404 returned error can't find the container with id 61d0c3b8f04f98e80718368264e97e47b63482552a3ddae2f6e5adb4dc3af1dc Jul 16 13:32:18.996013 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:18.995980 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" event={"ID":"06e85349-db65-42c9-b5c6-ae14b9de175d","Type":"ContainerStarted","Data":"61d0c3b8f04f98e80718368264e97e47b63482552a3ddae2f6e5adb4dc3af1dc"} Jul 16 13:32:20.003849 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:20.003811 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" event={"ID":"06e85349-db65-42c9-b5c6-ae14b9de175d","Type":"ContainerStarted","Data":"aced9b7c053ba7277ef0ecd27bae9666ce7864b5fe642be6f1da095f9639ea6f"} Jul 16 13:32:20.026880 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:20.026831 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" podStartSLOduration=2.026814655 podStartE2EDuration="2.026814655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:32:20.026187844 +0000 UTC m=+2848.021328623" watchObservedRunningTime="2026-07-16 13:32:20.026814655 +0000 UTC m=+2848.021955414" Jul 16 13:32:28.789358 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:28.789323 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:28.789900 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:28.789381 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:28.801720 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:28.801698 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:32:29.066945 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:32:29.066856 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:33:14.173157 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:14.173074 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d"] Jul 16 13:33:27.232511 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:27.232477 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-140680305587520"] Jul 16 13:33:27.235925 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:27.235907 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:27.238736 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:27.238715 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fd4h6\"" Jul 16 13:33:27.245420 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:27.245395 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140680305587520"] Jul 16 13:33:27.297016 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:27.296985 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqfpm\" (UniqueName: \"kubernetes.io/projected/31647298-2104-44d1-a12b-17c89da36f87-kube-api-access-gqfpm\") pod \"test-curl-7614-140680305587520\" (UID: \"31647298-2104-44d1-a12b-17c89da36f87\") " pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:27.397956 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:27.397919 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqfpm\" (UniqueName: \"kubernetes.io/projected/31647298-2104-44d1-a12b-17c89da36f87-kube-api-access-gqfpm\") pod \"test-curl-7614-140680305587520\" (UID: \"31647298-2104-44d1-a12b-17c89da36f87\") " pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:27.408605 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:27.408580 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqfpm\" (UniqueName: \"kubernetes.io/projected/31647298-2104-44d1-a12b-17c89da36f87-kube-api-access-gqfpm\") pod \"test-curl-7614-140680305587520\" (UID: \"31647298-2104-44d1-a12b-17c89da36f87\") " pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:27.545365 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:27.545331 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:27.688471 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:27.688443 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140680305587520"] Jul 16 13:33:27.690612 ip-10-0-138-198 kubenswrapper[2565]: W0716 13:33:27.690584 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31647298_2104_44d1_a12b_17c89da36f87.slice/crio-fe6a8d7ee59c3de4ddba97fbb0493f1d80f70ca9241c258ac774122331a66087 WatchSource:0}: Error finding container fe6a8d7ee59c3de4ddba97fbb0493f1d80f70ca9241c258ac774122331a66087: Status 404 returned error can't find the container with id fe6a8d7ee59c3de4ddba97fbb0493f1d80f70ca9241c258ac774122331a66087 Jul 16 13:33:27.691923 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:27.691906 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 13:33:28.398594 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:28.398563 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140680305587520" event={"ID":"31647298-2104-44d1-a12b-17c89da36f87","Type":"ContainerStarted","Data":"fe6a8d7ee59c3de4ddba97fbb0493f1d80f70ca9241c258ac774122331a66087"} Jul 16 13:33:29.193251 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.193204 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" podUID="06e85349-db65-42c9-b5c6-ae14b9de175d" containerName="main" containerID="cri-o://aced9b7c053ba7277ef0ecd27bae9666ce7864b5fe642be6f1da095f9639ea6f" gracePeriod=45 Jul 16 13:33:29.407878 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.407841 2565 generic.go:356] "Generic (PLEG): container finished" podID="06e85349-db65-42c9-b5c6-ae14b9de175d" containerID="aced9b7c053ba7277ef0ecd27bae9666ce7864b5fe642be6f1da095f9639ea6f" exitCode=0 Jul 16 13:33:29.408283 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.408015 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" event={"ID":"06e85349-db65-42c9-b5c6-ae14b9de175d","Type":"ContainerDied","Data":"aced9b7c053ba7277ef0ecd27bae9666ce7864b5fe642be6f1da095f9639ea6f"} Jul 16 13:33:29.409880 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.409853 2565 generic.go:356] "Generic (PLEG): container finished" podID="31647298-2104-44d1-a12b-17c89da36f87" containerID="338a4463cb31e5a9b6226a55334888822401576ad4013e74c745a202ad73b22f" exitCode=0 Jul 16 13:33:29.409997 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.409908 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140680305587520" event={"ID":"31647298-2104-44d1-a12b-17c89da36f87","Type":"ContainerDied","Data":"338a4463cb31e5a9b6226a55334888822401576ad4013e74c745a202ad73b22f"} Jul 16 13:33:29.459947 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.459924 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:33:29.518636 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.518603 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-tmp-dir\") pod \"06e85349-db65-42c9-b5c6-ae14b9de175d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " Jul 16 13:33:29.518826 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.518654 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-dshm\") pod \"06e85349-db65-42c9-b5c6-ae14b9de175d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " Jul 16 13:33:29.518826 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.518707 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e85349-db65-42c9-b5c6-ae14b9de175d-tls-certs\") pod \"06e85349-db65-42c9-b5c6-ae14b9de175d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " Jul 16 13:33:29.518826 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.518745 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-model-cache\") pod \"06e85349-db65-42c9-b5c6-ae14b9de175d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " Jul 16 13:33:29.518826 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.518794 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vhjb\" (UniqueName: \"kubernetes.io/projected/06e85349-db65-42c9-b5c6-ae14b9de175d-kube-api-access-2vhjb\") pod \"06e85349-db65-42c9-b5c6-ae14b9de175d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " Jul 16 13:33:29.519050 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.518850 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-home\") pod \"06e85349-db65-42c9-b5c6-ae14b9de175d\" (UID: \"06e85349-db65-42c9-b5c6-ae14b9de175d\") " Jul 16 13:33:29.519050 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.518918 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06e85349-db65-42c9-b5c6-ae14b9de175d" (UID: "06e85349-db65-42c9-b5c6-ae14b9de175d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:33:29.519050 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.519038 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-model-cache" (OuterVolumeSpecName: "model-cache") pod "06e85349-db65-42c9-b5c6-ae14b9de175d" (UID: "06e85349-db65-42c9-b5c6-ae14b9de175d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:33:29.519218 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.519195 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-model-cache\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:33:29.519369 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.519223 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-tmp-dir\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:33:29.519369 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.519194 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-home" (OuterVolumeSpecName: "home") pod "06e85349-db65-42c9-b5c6-ae14b9de175d" (UID: "06e85349-db65-42c9-b5c6-ae14b9de175d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:33:29.521028 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.520998 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-dshm" (OuterVolumeSpecName: "dshm") pod "06e85349-db65-42c9-b5c6-ae14b9de175d" (UID: "06e85349-db65-42c9-b5c6-ae14b9de175d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:33:29.521028 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.520999 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e85349-db65-42c9-b5c6-ae14b9de175d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06e85349-db65-42c9-b5c6-ae14b9de175d" (UID: "06e85349-db65-42c9-b5c6-ae14b9de175d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:33:29.521167 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.521067 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e85349-db65-42c9-b5c6-ae14b9de175d-kube-api-access-2vhjb" (OuterVolumeSpecName: "kube-api-access-2vhjb") pod "06e85349-db65-42c9-b5c6-ae14b9de175d" (UID: "06e85349-db65-42c9-b5c6-ae14b9de175d"). InnerVolumeSpecName "kube-api-access-2vhjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:33:29.620788 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.620753 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-dshm\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:33:29.620788 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.620781 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e85349-db65-42c9-b5c6-ae14b9de175d-tls-certs\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:33:29.620788 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.620795 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vhjb\" (UniqueName: \"kubernetes.io/projected/06e85349-db65-42c9-b5c6-ae14b9de175d-kube-api-access-2vhjb\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:33:29.621051 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:29.620806 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e85349-db65-42c9-b5c6-ae14b9de175d-home\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:33:30.417342 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:30.417308 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" Jul 16 13:33:30.417342 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:30.417324 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d" event={"ID":"06e85349-db65-42c9-b5c6-ae14b9de175d","Type":"ContainerDied","Data":"61d0c3b8f04f98e80718368264e97e47b63482552a3ddae2f6e5adb4dc3af1dc"} Jul 16 13:33:30.417855 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:30.417373 2565 scope.go:117] "RemoveContainer" containerID="aced9b7c053ba7277ef0ecd27bae9666ce7864b5fe642be6f1da095f9639ea6f" Jul 16 13:33:30.442309 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:30.442280 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d"] Jul 16 13:33:30.447019 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:30.446995 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/rate-test-model-c7dd05-kserve-5b46985547-n7r7d"] Jul 16 13:33:30.547274 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:30.547223 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06e85349-db65-42c9-b5c6-ae14b9de175d" path="/var/lib/kubelet/pods/06e85349-db65-42c9-b5c6-ae14b9de175d/volumes" Jul 16 13:33:30.556388 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:30.556365 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:30.602646 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:30.602621 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7614-140680305587520_31647298-2104-44d1-a12b-17c89da36f87/test-curl-7614-140680305587520/0.log" Jul 16 13:33:30.630972 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:30.630943 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-140680305587520"] Jul 16 13:33:30.637979 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:30.637954 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqfpm\" (UniqueName: \"kubernetes.io/projected/31647298-2104-44d1-a12b-17c89da36f87-kube-api-access-gqfpm\") pod \"31647298-2104-44d1-a12b-17c89da36f87\" (UID: \"31647298-2104-44d1-a12b-17c89da36f87\") " Jul 16 13:33:30.638389 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:30.638356 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-140680305587520"] Jul 16 13:33:30.640303 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:30.640284 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31647298-2104-44d1-a12b-17c89da36f87-kube-api-access-gqfpm" (OuterVolumeSpecName: "kube-api-access-gqfpm") pod "31647298-2104-44d1-a12b-17c89da36f87" (UID: "31647298-2104-44d1-a12b-17c89da36f87"). InnerVolumeSpecName "kube-api-access-gqfpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:33:30.738973 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:30.738885 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqfpm\" (UniqueName: \"kubernetes.io/projected/31647298-2104-44d1-a12b-17c89da36f87-kube-api-access-gqfpm\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:33:31.425422 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:31.425396 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:31.425777 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:31.425424 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe6a8d7ee59c3de4ddba97fbb0493f1d80f70ca9241c258ac774122331a66087" Jul 16 13:33:31.435189 ip-10-0-138-198 kubenswrapper[2565]: E0716 13:33:31.435163 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-140680305587520\" is forbidden: User \"system:node:ip-10-0-138-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-198.ec2.internal' and this object" podUID="31647298-2104-44d1-a12b-17c89da36f87" pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:32.545188 ip-10-0-138-198 kubenswrapper[2565]: E0716 13:33:32.545147 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-140680305587520\" is forbidden: User \"system:node:ip-10-0-138-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-198.ec2.internal' and this object" podUID="31647298-2104-44d1-a12b-17c89da36f87" pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:32.546407 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:33:32.546387 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31647298-2104-44d1-a12b-17c89da36f87" path="/var/lib/kubelet/pods/31647298-2104-44d1-a12b-17c89da36f87/volumes" Jul 16 13:34:25.454214 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:25.449149 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9"] Jul 16 13:34:25.454214 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:25.449526 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" podUID="e3bb0969-5576-481a-b2d5-60a08a9d39de" containerName="istio-proxy" containerID="cri-o://d21888187d1675d39a9d7018415d6b63e11c52afc08ce03c9b0c235105ff70a4" gracePeriod=30 Jul 16 13:34:30.713981 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.713957 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:34:30.780919 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.780884 2565 generic.go:356] "Generic (PLEG): container finished" podID="e3bb0969-5576-481a-b2d5-60a08a9d39de" containerID="d21888187d1675d39a9d7018415d6b63e11c52afc08ce03c9b0c235105ff70a4" exitCode=0 Jul 16 13:34:30.781115 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.781092 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" event={"ID":"e3bb0969-5576-481a-b2d5-60a08a9d39de","Type":"ContainerDied","Data":"d21888187d1675d39a9d7018415d6b63e11c52afc08ce03c9b0c235105ff70a4"} Jul 16 13:34:30.781278 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.781174 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" event={"ID":"e3bb0969-5576-481a-b2d5-60a08a9d39de","Type":"ContainerDied","Data":"e72e380de8708eec8b1b7f9b0ded7659dcf5aa0c77076a1969764d7a596e20d9"} Jul 16 13:34:30.781278 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.781213 2565 scope.go:117] "RemoveContainer" containerID="d21888187d1675d39a9d7018415d6b63e11c52afc08ce03c9b0c235105ff70a4" Jul 16 13:34:30.781492 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.781449 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9" Jul 16 13:34:30.791197 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.791176 2565 scope.go:117] "RemoveContainer" containerID="e8599fd23d7a0f21cb9cdcb7074874b98048858adf998f6cad30ba4e3bd4d8fe" Jul 16 13:34:30.801552 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.801528 2565 scope.go:117] "RemoveContainer" containerID="d21888187d1675d39a9d7018415d6b63e11c52afc08ce03c9b0c235105ff70a4" Jul 16 13:34:30.802012 ip-10-0-138-198 kubenswrapper[2565]: E0716 13:34:30.801986 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21888187d1675d39a9d7018415d6b63e11c52afc08ce03c9b0c235105ff70a4\": container with ID starting with d21888187d1675d39a9d7018415d6b63e11c52afc08ce03c9b0c235105ff70a4 not found: ID does not exist" containerID="d21888187d1675d39a9d7018415d6b63e11c52afc08ce03c9b0c235105ff70a4" Jul 16 13:34:30.802101 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.802022 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21888187d1675d39a9d7018415d6b63e11c52afc08ce03c9b0c235105ff70a4"} err="failed to get container status \"d21888187d1675d39a9d7018415d6b63e11c52afc08ce03c9b0c235105ff70a4\": rpc error: code = NotFound desc = could not find container \"d21888187d1675d39a9d7018415d6b63e11c52afc08ce03c9b0c235105ff70a4\": container with ID starting with d21888187d1675d39a9d7018415d6b63e11c52afc08ce03c9b0c235105ff70a4 not found: ID does not exist" Jul 16 13:34:30.802101 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.802048 2565 scope.go:117] "RemoveContainer" containerID="e8599fd23d7a0f21cb9cdcb7074874b98048858adf998f6cad30ba4e3bd4d8fe" Jul 16 13:34:30.802326 ip-10-0-138-198 kubenswrapper[2565]: E0716 13:34:30.802307 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8599fd23d7a0f21cb9cdcb7074874b98048858adf998f6cad30ba4e3bd4d8fe\": container with ID starting with e8599fd23d7a0f21cb9cdcb7074874b98048858adf998f6cad30ba4e3bd4d8fe not found: ID does not exist" containerID="e8599fd23d7a0f21cb9cdcb7074874b98048858adf998f6cad30ba4e3bd4d8fe" Jul 16 13:34:30.802382 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.802333 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8599fd23d7a0f21cb9cdcb7074874b98048858adf998f6cad30ba4e3bd4d8fe"} err="failed to get container status \"e8599fd23d7a0f21cb9cdcb7074874b98048858adf998f6cad30ba4e3bd4d8fe\": rpc error: code = NotFound desc = could not find container \"e8599fd23d7a0f21cb9cdcb7074874b98048858adf998f6cad30ba4e3bd4d8fe\": container with ID starting with e8599fd23d7a0f21cb9cdcb7074874b98048858adf998f6cad30ba4e3bd4d8fe not found: ID does not exist" Jul 16 13:34:30.810580 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.810564 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-envoy\") pod \"e3bb0969-5576-481a-b2d5-60a08a9d39de\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " Jul 16 13:34:30.810652 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.810605 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bb0969-5576-481a-b2d5-60a08a9d39de-istiod-ca-cert\") pod \"e3bb0969-5576-481a-b2d5-60a08a9d39de\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " Jul 16 13:34:30.810689 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.810672 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-podinfo\") pod \"e3bb0969-5576-481a-b2d5-60a08a9d39de\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " Jul 16 13:34:30.810722 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.810697 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-workload-socket\") pod \"e3bb0969-5576-481a-b2d5-60a08a9d39de\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " Jul 16 13:34:30.810756 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.810719 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-credential-socket\") pod \"e3bb0969-5576-481a-b2d5-60a08a9d39de\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " Jul 16 13:34:30.810805 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.810764 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-token\") pod \"e3bb0969-5576-481a-b2d5-60a08a9d39de\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " Jul 16 13:34:30.810805 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.810792 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-data\") pod \"e3bb0969-5576-481a-b2d5-60a08a9d39de\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " Jul 16 13:34:30.810899 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.810825 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-workload-certs\") pod \"e3bb0969-5576-481a-b2d5-60a08a9d39de\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " Jul 16 13:34:30.810899 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.810863 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq5hs\" (UniqueName: \"kubernetes.io/projected/e3bb0969-5576-481a-b2d5-60a08a9d39de-kube-api-access-vq5hs\") pod \"e3bb0969-5576-481a-b2d5-60a08a9d39de\" (UID: \"e3bb0969-5576-481a-b2d5-60a08a9d39de\") " Jul 16 13:34:30.811031 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.811005 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3bb0969-5576-481a-b2d5-60a08a9d39de" (UID: "e3bb0969-5576-481a-b2d5-60a08a9d39de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:34:30.811271 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.811119 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3bb0969-5576-481a-b2d5-60a08a9d39de" (UID: "e3bb0969-5576-481a-b2d5-60a08a9d39de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:34:30.811428 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.811130 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3bb0969-5576-481a-b2d5-60a08a9d39de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3bb0969-5576-481a-b2d5-60a08a9d39de" (UID: "e3bb0969-5576-481a-b2d5-60a08a9d39de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:34:30.811517 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.811350 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3bb0969-5576-481a-b2d5-60a08a9d39de" (UID: "e3bb0969-5576-481a-b2d5-60a08a9d39de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:34:30.811774 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.811748 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3bb0969-5576-481a-b2d5-60a08a9d39de" (UID: "e3bb0969-5576-481a-b2d5-60a08a9d39de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:34:30.813182 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.813160 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3bb0969-5576-481a-b2d5-60a08a9d39de" (UID: "e3bb0969-5576-481a-b2d5-60a08a9d39de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:34:30.813361 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.813338 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3bb0969-5576-481a-b2d5-60a08a9d39de" (UID: "e3bb0969-5576-481a-b2d5-60a08a9d39de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 13:34:30.813426 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.813338 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3bb0969-5576-481a-b2d5-60a08a9d39de" (UID: "e3bb0969-5576-481a-b2d5-60a08a9d39de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:34:30.813588 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.813569 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bb0969-5576-481a-b2d5-60a08a9d39de-kube-api-access-vq5hs" (OuterVolumeSpecName: "kube-api-access-vq5hs") pod "e3bb0969-5576-481a-b2d5-60a08a9d39de" (UID: "e3bb0969-5576-481a-b2d5-60a08a9d39de"). InnerVolumeSpecName "kube-api-access-vq5hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:34:30.911917 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.911888 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bb0969-5576-481a-b2d5-60a08a9d39de-istiod-ca-cert\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:34:30.911917 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.911912 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-podinfo\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:34:30.911917 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.911921 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-workload-socket\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:34:30.912129 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.911929 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-credential-socket\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:34:30.912129 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.911938 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-token\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:34:30.912129 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.911947 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-data\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:34:30.912129 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.911955 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-workload-certs\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:34:30.912129 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.911964 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq5hs\" (UniqueName: \"kubernetes.io/projected/e3bb0969-5576-481a-b2d5-60a08a9d39de-kube-api-access-vq5hs\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:34:30.912129 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:30.911972 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bb0969-5576-481a-b2d5-60a08a9d39de-istio-envoy\") on node \"ip-10-0-138-198.ec2.internal\" DevicePath \"\"" Jul 16 13:34:31.110001 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:31.109971 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9"] Jul 16 13:34:31.114978 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:31.114955 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-c7dd05-openshift-default-8484969b7-6tgn9"] Jul 16 13:34:32.547255 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:32.547193 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3bb0969-5576-481a-b2d5-60a08a9d39de" path="/var/lib/kubelet/pods/e3bb0969-5576-481a-b2d5-60a08a9d39de/volumes" Jul 16 13:34:32.592343 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:32.592319 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9d668944c-n5ft8_3277b759-4517-4cb2-a78b-108ca160674d/copy-manifests/0.log" Jul 16 13:34:32.602797 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:32.602775 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9d668944c-n5ft8_3277b759-4517-4cb2-a78b-108ca160674d/manager/0.log" Jul 16 13:34:33.744986 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:33.744951 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-62a003-v2fgs_180f2e64-dad0-430a-948f-21fa3e82cca0/revoke-keys/0.log" Jul 16 13:34:33.867063 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:33.867035 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-c92f28-cb9kv_f39e2b07-cea0-46f6-b398-a39e3a901445/revoke-keys/0.log" Jul 16 13:34:34.122922 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:34.122888 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-00668a-rxdj9_5e1c2b14-07f4-4150-8bbf-6114e3498ebc/revoke-keys/0.log" Jul 16 13:34:34.247369 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:34.247342 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-ab35b1-64ljq_f83dcf67-ae55-40a6-8883-d624b7050ed5/revoke-keys/0.log" Jul 16 13:34:35.810012 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:35.809978 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh_f4b68c66-4998-486a-8c72-6a0378055557/util/0.log" Jul 16 13:34:35.824286 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:35.824259 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh_f4b68c66-4998-486a-8c72-6a0378055557/pull/0.log" Jul 16 13:34:35.832622 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:35.832597 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh_f4b68c66-4998-486a-8c72-6a0378055557/extract/0.log" Jul 16 13:34:35.952495 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:35.952452 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh_00257efc-a528-46ae-8755-3ced72e384be/pull/0.log" Jul 16 13:34:35.964527 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:35.964497 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh_00257efc-a528-46ae-8755-3ced72e384be/extract/0.log" Jul 16 13:34:35.982665 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:35.982635 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh_00257efc-a528-46ae-8755-3ced72e384be/util/0.log" Jul 16 13:34:36.133179 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:36.133093 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m_efb077ca-6723-44fc-afee-ada3a8f7f000/util/0.log" Jul 16 13:34:36.142615 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:36.142592 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m_efb077ca-6723-44fc-afee-ada3a8f7f000/pull/0.log" Jul 16 13:34:36.150898 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:36.150879 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m_efb077ca-6723-44fc-afee-ada3a8f7f000/extract/0.log" Jul 16 13:34:36.275683 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:36.275654 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp_ccf6bee9-d871-4959-baa4-ec2872dde4a3/pull/0.log" Jul 16 13:34:36.289546 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:36.289518 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp_ccf6bee9-d871-4959-baa4-ec2872dde4a3/extract/0.log" Jul 16 13:34:36.298945 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:36.298921 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp_ccf6bee9-d871-4959-baa4-ec2872dde4a3/util/0.log" Jul 16 13:34:36.558672 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:36.556075 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-w9d5v_58dca71d-bb6d-406f-9699-fe45c5d9bbed/manager/0.log" Jul 16 13:34:36.691662 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:36.691630 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-dbjqw_a7947a67-b83e-4e01-941c-56704482046b/manager/0.log" Jul 16 13:34:36.949800 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:36.949712 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-8c7pg_a40b7195-74d8-4613-8409-53054056fdb2/registry-server/0.log" Jul 16 13:34:37.354146 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:37.354117 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-l54v4_477a480a-9ea5-46db-9886-880006f6f9d0/manager/0.log" Jul 16 13:34:37.738535 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:37.738462 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf44r4nm_1dc9d367-ab3a-4432-b7f1-47a17d8f0c89/istio-proxy/0.log" Jul 16 13:34:38.234071 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:38.234029 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nwcbm_0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb/istio-proxy/1.log" Jul 16 13:34:38.618820 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:38.618787 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75f7d966c4-bb2lf_15790d2d-298f-4fff-9945-c8bc693a04b1/router/0.log" Jul 16 13:34:39.107530 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:39.107497 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-pmpq2_84c7fb18-47a7-4e95-a8a7-74607faacae8/main/0.log" Jul 16 13:34:39.247616 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:39.247584 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-w2t78_09ddee62-e0ae-406f-aff9-b9089de0123e/main/0.log" Jul 16 13:34:39.398031 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:39.397950 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzbjr8_356e28f2-b8fc-4634-bc08-27edf9f46bfe/main/0.log" Jul 16 13:34:49.296737 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:49.296709 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zdrc5_5de8db0d-cb13-482f-b0f6-8259d3cb37f9/global-pull-secret-syncer/0.log" Jul 16 13:34:49.463028 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:49.462996 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v2l9k_34601419-53fc-4175-b4b9-44de20f22277/konnectivity-agent/0.log" Jul 16 13:34:49.540143 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:49.540118 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-198.ec2.internal_6594cb49ca46862b517c534a4f67f101/haproxy/0.log" Jul 16 13:34:52.776462 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:52.776433 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 13:34:52.786872 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:52.786853 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4bdr_56aed7de-c338-4c39-98d3-a6e43b58d9b7/ovn-acl-logging/0.log" Jul 16 13:34:54.103380 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:54.103348 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh_f4b68c66-4998-486a-8c72-6a0378055557/extract/0.log" Jul 16 13:34:54.146104 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:54.146077 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh_f4b68c66-4998-486a-8c72-6a0378055557/util/0.log" Jul 16 13:34:54.195122 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:54.195096 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595jzzh_f4b68c66-4998-486a-8c72-6a0378055557/pull/0.log" Jul 16 13:34:54.239867 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:54.239844 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh_00257efc-a528-46ae-8755-3ced72e384be/extract/0.log" Jul 16 13:34:54.278915 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:54.278892 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh_00257efc-a528-46ae-8755-3ced72e384be/util/0.log" Jul 16 13:34:54.314331 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:54.314312 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7ndh_00257efc-a528-46ae-8755-3ced72e384be/pull/0.log" Jul 16 13:34:54.356554 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:54.356475 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m_efb077ca-6723-44fc-afee-ada3a8f7f000/extract/0.log" Jul 16 13:34:54.409858 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:54.409832 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m_efb077ca-6723-44fc-afee-ada3a8f7f000/util/0.log" Jul 16 13:34:54.461508 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:54.461479 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rxv7m_efb077ca-6723-44fc-afee-ada3a8f7f000/pull/0.log" Jul 16 13:34:54.503977 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:54.503942 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp_ccf6bee9-d871-4959-baa4-ec2872dde4a3/extract/0.log" Jul 16 13:34:54.538584 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:54.538558 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp_ccf6bee9-d871-4959-baa4-ec2872dde4a3/util/0.log" Jul 16 13:34:54.575835 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:54.575797 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vs5tp_ccf6bee9-d871-4959-baa4-ec2872dde4a3/pull/0.log" Jul 16 13:34:54.942157 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:54.942125 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-w9d5v_58dca71d-bb6d-406f-9699-fe45c5d9bbed/manager/0.log" Jul 16 13:34:54.981407 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:54.981378 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-dbjqw_a7947a67-b83e-4e01-941c-56704482046b/manager/0.log" Jul 16 13:34:55.079107 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:55.079082 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-8c7pg_a40b7195-74d8-4613-8409-53054056fdb2/registry-server/0.log" Jul 16 13:34:55.341199 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:55.341173 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-l54v4_477a480a-9ea5-46db-9886-880006f6f9d0/manager/0.log" Jul 16 13:34:57.316563 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:57.316532 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-c7ng7_7a6c1393-8665-4d4a-9238-77f24e498e50/cluster-monitoring-operator/0.log" Jul 16 13:34:57.539267 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:57.539221 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f4965c6b-jppmp_80882f06-3d35-44aa-8629-f40dc451f7aa/metrics-server/0.log" Jul 16 13:34:57.605964 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:57.605891 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-xvmgv_2cb7eec3-02a8-4a53-ac8d-af061fde4a9c/monitoring-plugin/0.log" Jul 16 13:34:57.678741 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:57.678718 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c6z6g_3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b/node-exporter/0.log" Jul 16 13:34:57.781539 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:57.781515 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c6z6g_3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b/kube-rbac-proxy/0.log" Jul 16 13:34:57.862030 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:57.861963 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c6z6g_3bd1c8e4-231a-4a42-88c3-9f5d8c653f7b/init-textfile/0.log" Jul 16 13:34:58.309307 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:58.309282 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-p6swb_c5280694-1df9-4c77-ac14-eee54b7cc27f/kube-rbac-proxy-main/0.log" Jul 16 13:34:58.374877 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:58.374851 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-p6swb_c5280694-1df9-4c77-ac14-eee54b7cc27f/kube-rbac-proxy-self/0.log" Jul 16 13:34:58.441088 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:58.441062 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-p6swb_c5280694-1df9-4c77-ac14-eee54b7cc27f/openshift-state-metrics/0.log" Jul 16 13:34:59.023488 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.023457 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-hmrt5_acbfde83-0959-4f8e-a53e-33c800463dec/prometheus-operator/0.log" Jul 16 13:34:59.099088 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.099057 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-hmrt5_acbfde83-0959-4f8e-a53e-33c800463dec/kube-rbac-proxy/0.log" Jul 16 13:34:59.174973 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.174945 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-8lnf7_d2fafe97-2a61-4045-925c-89129f1c1f72/prometheus-operator-admission-webhook/0.log" Jul 16 13:34:59.671658 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.671626 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp"] Jul 16 13:34:59.672022 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.671973 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06e85349-db65-42c9-b5c6-ae14b9de175d" containerName="main" Jul 16 13:34:59.672022 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.671984 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="06e85349-db65-42c9-b5c6-ae14b9de175d" containerName="main" Jul 16 13:34:59.672022 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.672007 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31647298-2104-44d1-a12b-17c89da36f87" containerName="test-curl-7614-140680305587520" Jul 16 13:34:59.672022 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.672012 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="31647298-2104-44d1-a12b-17c89da36f87" containerName="test-curl-7614-140680305587520" Jul 16 13:34:59.672022 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.672020 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3bb0969-5576-481a-b2d5-60a08a9d39de" containerName="istio-proxy" Jul 16 13:34:59.672270 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.672027 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3bb0969-5576-481a-b2d5-60a08a9d39de" containerName="istio-proxy" Jul 16 13:34:59.672270 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.672033 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3bb0969-5576-481a-b2d5-60a08a9d39de" containerName="istio-proxy" Jul 16 13:34:59.672270 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.672039 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3bb0969-5576-481a-b2d5-60a08a9d39de" containerName="istio-proxy" Jul 16 13:34:59.672270 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.672094 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3bb0969-5576-481a-b2d5-60a08a9d39de" containerName="istio-proxy" Jul 16 13:34:59.672270 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.672103 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3bb0969-5576-481a-b2d5-60a08a9d39de" containerName="istio-proxy" Jul 16 13:34:59.672270 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.672111 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="31647298-2104-44d1-a12b-17c89da36f87" containerName="test-curl-7614-140680305587520" Jul 16 13:34:59.672270 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.672116 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="06e85349-db65-42c9-b5c6-ae14b9de175d" containerName="main" Jul 16 13:34:59.676604 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.676586 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:34:59.683313 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.683090 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ljjhv\"/\"default-dockercfg-t5vwb\"" Jul 16 13:34:59.683396 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.683305 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ljjhv\"/\"openshift-service-ca.crt\"" Jul 16 13:34:59.696784 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.696766 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ljjhv\"/\"kube-root-ca.crt\"" Jul 16 13:34:59.742592 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.742565 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp"] Jul 16 13:34:59.771390 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.771355 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6310e48b-3168-408c-963e-8a9f964bf844-proc\") pod \"perf-node-gather-daemonset-rkznp\" (UID: \"6310e48b-3168-408c-963e-8a9f964bf844\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:34:59.771541 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.771396 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6310e48b-3168-408c-963e-8a9f964bf844-podres\") pod \"perf-node-gather-daemonset-rkznp\" (UID: \"6310e48b-3168-408c-963e-8a9f964bf844\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:34:59.771541 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.771442 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6310e48b-3168-408c-963e-8a9f964bf844-lib-modules\") pod \"perf-node-gather-daemonset-rkznp\" (UID: \"6310e48b-3168-408c-963e-8a9f964bf844\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:34:59.771541 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.771534 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6310e48b-3168-408c-963e-8a9f964bf844-sys\") pod \"perf-node-gather-daemonset-rkznp\" (UID: \"6310e48b-3168-408c-963e-8a9f964bf844\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:34:59.771667 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.771551 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvp4k\" (UniqueName: \"kubernetes.io/projected/6310e48b-3168-408c-963e-8a9f964bf844-kube-api-access-pvp4k\") pod \"perf-node-gather-daemonset-rkznp\" (UID: \"6310e48b-3168-408c-963e-8a9f964bf844\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:34:59.872149 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.872115 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6310e48b-3168-408c-963e-8a9f964bf844-sys\") pod \"perf-node-gather-daemonset-rkznp\" (UID: \"6310e48b-3168-408c-963e-8a9f964bf844\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:34:59.872149 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.872151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvp4k\" (UniqueName: \"kubernetes.io/projected/6310e48b-3168-408c-963e-8a9f964bf844-kube-api-access-pvp4k\") pod \"perf-node-gather-daemonset-rkznp\" (UID: \"6310e48b-3168-408c-963e-8a9f964bf844\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:34:59.872392 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.872219 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6310e48b-3168-408c-963e-8a9f964bf844-proc\") pod \"perf-node-gather-daemonset-rkznp\" (UID: \"6310e48b-3168-408c-963e-8a9f964bf844\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:34:59.872392 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.872270 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6310e48b-3168-408c-963e-8a9f964bf844-sys\") pod \"perf-node-gather-daemonset-rkznp\" (UID: \"6310e48b-3168-408c-963e-8a9f964bf844\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:34:59.872392 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.872279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6310e48b-3168-408c-963e-8a9f964bf844-podres\") pod \"perf-node-gather-daemonset-rkznp\" (UID: \"6310e48b-3168-408c-963e-8a9f964bf844\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:34:59.872392 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.872327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6310e48b-3168-408c-963e-8a9f964bf844-lib-modules\") pod \"perf-node-gather-daemonset-rkznp\" (UID: \"6310e48b-3168-408c-963e-8a9f964bf844\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:34:59.872392 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.872346 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6310e48b-3168-408c-963e-8a9f964bf844-proc\") pod \"perf-node-gather-daemonset-rkznp\" (UID: \"6310e48b-3168-408c-963e-8a9f964bf844\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:34:59.872555 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.872395 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6310e48b-3168-408c-963e-8a9f964bf844-podres\") pod \"perf-node-gather-daemonset-rkznp\" (UID: \"6310e48b-3168-408c-963e-8a9f964bf844\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:34:59.872555 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.872468 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6310e48b-3168-408c-963e-8a9f964bf844-lib-modules\") pod \"perf-node-gather-daemonset-rkznp\" (UID: \"6310e48b-3168-408c-963e-8a9f964bf844\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:34:59.889112 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.889082 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvp4k\" (UniqueName: \"kubernetes.io/projected/6310e48b-3168-408c-963e-8a9f964bf844-kube-api-access-pvp4k\") pod \"perf-node-gather-daemonset-rkznp\" (UID: \"6310e48b-3168-408c-963e-8a9f964bf844\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:34:59.986140 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:34:59.986065 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:35:00.147763 ip-10-0-138-198 kubenswrapper[2565]: W0716 13:35:00.147726 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6310e48b_3168_408c_963e_8a9f964bf844.slice/crio-f6910e05de9d8acff5c08832e5ebb7ff062c202fdea05f624064925c62eb9cb9 WatchSource:0}: Error finding container f6910e05de9d8acff5c08832e5ebb7ff062c202fdea05f624064925c62eb9cb9: Status 404 returned error can't find the container with id f6910e05de9d8acff5c08832e5ebb7ff062c202fdea05f624064925c62eb9cb9 Jul 16 13:35:00.156795 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:00.156773 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp"] Jul 16 13:35:00.953647 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:00.953610 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" event={"ID":"6310e48b-3168-408c-963e-8a9f964bf844","Type":"ContainerStarted","Data":"5cc96c68b4f1f9dc9119df97f90a6544ec7384771ac11edffc2805c4d9368d50"} Jul 16 13:35:00.954090 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:00.953665 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" event={"ID":"6310e48b-3168-408c-963e-8a9f964bf844","Type":"ContainerStarted","Data":"f6910e05de9d8acff5c08832e5ebb7ff062c202fdea05f624064925c62eb9cb9"} Jul 16 13:35:00.954090 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:00.953697 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:35:00.981111 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:00.981053 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" podStartSLOduration=1.981036925 podStartE2EDuration="1.981036925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 13:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:35:00.979749208 +0000 UTC m=+3008.974889971" watchObservedRunningTime="2026-07-16 13:35:00.981036925 +0000 UTC m=+3008.976177710" Jul 16 13:35:02.565082 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:02.565055 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5549db9dd8-z4wts_3d385d95-ed54-4afa-ab9d-953afbb57397/console/0.log" Jul 16 13:35:02.634855 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:02.634822 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-gz4px_180b114f-387d-46cf-a493-7ad185105df3/download-server/0.log" Jul 16 13:35:04.946330 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:04.946301 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5dxp9_bd384a9f-61e7-4509-8346-5d345021a603/dns/0.log" Jul 16 13:35:05.017206 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:05.017181 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5dxp9_bd384a9f-61e7-4509-8346-5d345021a603/kube-rbac-proxy/0.log" Jul 16 13:35:05.436151 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:05.436123 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-smgrz_eb3b63d0-7a24-407c-891b-ca52cb88a27c/dns-node-resolver/0.log" Jul 16 13:35:06.268630 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:06.268603 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g64cf_e1549474-0299-489e-af96-767bdf28937a/node-ca/0.log" Jul 16 13:35:06.969973 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:06.969943 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-rkznp" Jul 16 13:35:07.571046 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:07.571018 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf44r4nm_1dc9d367-ab3a-4432-b7f1-47a17d8f0c89/istio-proxy/0.log" Jul 16 13:35:07.861038 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:07.860954 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nwcbm_0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb/istio-proxy/0.log" Jul 16 13:35:07.888182 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:07.888153 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nwcbm_0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb/istio-proxy/1.log" Jul 16 13:35:08.046009 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:08.045979 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75f7d966c4-bb2lf_15790d2d-298f-4fff-9945-c8bc693a04b1/router/0.log" Jul 16 13:35:09.030295 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:09.030263 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6n5wk_427c004b-ed80-4024-88ab-b817224073b4/serve-healthcheck-canary/0.log" Jul 16 13:35:09.430963 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:09.430934 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf44r4nm_1dc9d367-ab3a-4432-b7f1-47a17d8f0c89/istio-proxy/0.log" Jul 16 13:35:09.684619 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:09.684597 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nwcbm_0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb/istio-proxy/0.log" Jul 16 13:35:09.713326 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:09.713296 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nwcbm_0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb/istio-proxy/1.log" Jul 16 13:35:09.840729 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:09.840648 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75f7d966c4-bb2lf_15790d2d-298f-4fff-9945-c8bc693a04b1/router/0.log" Jul 16 13:35:09.895146 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:09.895112 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-qbhg8_73045a2a-1f69-48c1-96f1-54b941192680/insights-operator/0.log" Jul 16 13:35:09.897004 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:09.896981 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-qbhg8_73045a2a-1f69-48c1-96f1-54b941192680/insights-operator/1.log" Jul 16 13:35:10.308369 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:10.308334 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hxrcl_0031d292-78ee-4ea2-b31f-adb988b6d5e3/kube-rbac-proxy/0.log" Jul 16 13:35:10.351648 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:10.351626 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hxrcl_0031d292-78ee-4ea2-b31f-adb988b6d5e3/exporter/0.log" Jul 16 13:35:10.404705 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:10.404679 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hxrcl_0031d292-78ee-4ea2-b31f-adb988b6d5e3/extractor/0.log" Jul 16 13:35:11.025995 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:11.025944 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf44r4nm_1dc9d367-ab3a-4432-b7f1-47a17d8f0c89/istio-proxy/0.log" Jul 16 13:35:11.261120 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:11.261075 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nwcbm_0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb/istio-proxy/0.log" Jul 16 13:35:11.291033 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:11.290956 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nwcbm_0dd02b7f-5de8-46bb-b0ad-9dff6f0433eb/istio-proxy/1.log" Jul 16 13:35:11.450987 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:11.450959 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75f7d966c4-bb2lf_15790d2d-298f-4fff-9945-c8bc693a04b1/router/0.log" Jul 16 13:35:13.928197 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:13.928164 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9d668944c-n5ft8_3277b759-4517-4cb2-a78b-108ca160674d/manager/0.log" Jul 16 13:35:13.985921 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:13.985892 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9d668944c-n5ft8_3277b759-4517-4cb2-a78b-108ca160674d/copy-manifests/0.log" Jul 16 13:35:15.215273 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:15.215222 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5864f4f95b-fpwjz_bc2c5cdd-3a95-4ce6-afb3-0f3240733a75/manager/0.log" Jul 16 13:35:15.287525 ip-10-0-138-198 kubenswrapper[2565]: I0716 13:35:15.287495 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ln64s_8bd67b4b-b251-4f3b-92c2-a86695e53c8e/openshift-lws-operator/0.log"